Intermediate LevelPython
How does concurrent.futures.ThreadPoolExecutor simplify managing thread pools for concurrent I/O scraping tasks?
❓ Interview Prompt: Explain `ThreadPoolExecutor`, `executor.submit()`, `as_completed()`, Future objects, and managing concurrent I/O tasks.
'ThreadPoolExecutor' (from 'concurrent.futures') provides a modern high-level interface for managing worker thread pools:
Key Methods:
•
*'executor.submit(func, args)'**: Schedules callable and returns a 'Future' object.
•
'as_completed(futures)': Yields 'Future' instances as they complete in real time!
Code Implementation Example
python
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
def fetch_url(url):
time.sleep(0.1)
return f'Content of {url}'
urls = ['https://example.com/1', 'https://example.com/2', 'https://example.com/3']
with ThreadPoolExecutor(max_workers=5) as executor:
futures = [executor.submit(fetch_url, url) for url in urls]
for future in as_completed(futures):
print(future.result())💡Executing concurrent I/O tasks using ThreadPoolExecutor and as_completed.
🎯 Interview Key Takeaway
Use ThreadPoolExecutor for concurrent I/O-bound tasks like web scraping and network API calls.
#ThreadPoolExecutor#concurrent.futures#I/O#Intermediate
Back to all Python Questions→