Web scraping in Python usually starts with requests and Beautiful Soup. While this works well for static pages, it gets harder when a site loads content with JavaScript, blocks repeated requests, or changes its markup.
With the Olostep Python SDK, you can pass a URL and get the page back as markdown, HTML or plain text. Olostep renders the JavaScript and manages the browser and proxies for you, and this tutorial walks through your first scrapes. You'll learn how to:
- Set up the Olostep Python SDK and your API key
- Scrape a page to markdown
- Choose between markdown, HTML, and text output
- Scrape pages that load content with JavaScript
- Handle errors without stopping a run
All examples here scrape sites built for practice: Books to Scrape and Quotes to Scrape. So you can run every example as it's written here.
Just a note before we start: make sure you always scrape responsibly. The two practice sites let you get started with programmatically scrape the web. On other sites you wish to scrape, you'll need to check and follow their terms of service and robots.txt, collect only public data, and keep your request rate low.
Table of Contents
- Prerequisites
- How to Set Up the Client
- How to Scrape a Page to Markdown
- How to Choose an Output Format
- How to Scrape Pages That Load With JavaScript
- How to Handle Errors
- Conclusion
Prerequisites
- Python 3.11 to 3.13, which the Olostep SDK requires
- The Olostep SDK, installed with
pip install olostep - An Olostep API key from the dashboard. The first 500 credits are free. Also note: the docs price a scrape at 1 credit by default, and the code in this guide makes 4 scrapes, so running all the examples uses about 4 credits.
How to Set Up the Client
All the code in this guide uses one client, and the client needs your API key. The setup below reads the key from the OLOSTEP_API_KEY environment variable and asks for it when the variable is missing, so the key never sits in your code.
import os from getpass import getpass from olostep import Olostep api_key = os.getenv("OLOSTEP_API_KEY") or getpass("Olostep API key: ") client = Olostep(api_key=api_key) Let's now parse what each part does:
os.getenv("OLOSTEP_API_KEY")returns the key when you exported it, andNonewhen you didn't.getpassprompts for the key and hides what you type, which also works in a notebook such as Google Colab.Olostep(api_key=api_key)creates the client that all later code uses.
Run the later snippets in the same session, or paste the setup lines at the top of your file. Each snippet reuses this client. Keep keys out of files you commit.
How to Scrape a Page to Markdown
A scrape takes one URL and returns its content. Markdown is the default choice for most work, because it keeps the headings, lists, and tables of a page and drops the page chrome around them. This code scrapes the Books to Scrape home page.
page = client.scrapes.create( url_to_scrape="https://books.toscrape.com/", formats=["markdown"], ) print("Credits used:", page.credits_consumed) print(page.markdown_content[:400]) Here's what this code does:
url_to_scrapeis the page to read.formatslists the outputs you want. Each one arrives as an attribute on the result, such asmarkdown_content.credits_consumedreports what the call cost, so you can track spending as you go.- The result also carries
id,url, andretrieve_id, which identify the stored result.
Output:
Credits used: 1 All products | Books to Scrape - Sandbox [Books to Scrape](index.html) We love being scraped! * [Home](index.html) * All products * [Books](catalogue/category/books_1/index.html) * [Travel](catalogue/category/books/travel_2/index.html) * [Mystery](catalogue/category/books/mystery_3/index.html) * [Historical Fiction](catalogue/category/books/historical-fiction_ How to Choose an Output Format
One call can return several formats at once. This code asks for HTML, markdown, and text together and compares their sizes.
page = client.scrapes.create( url_to_scrape="https://books.toscrape.com/", formats=["html", "markdown", "text"], ) print("Credits used: ", page.credits_consumed) print("HTML characters: ", len(page.html_content)) print("Markdown characters:", len(page.markdown_content)) print("Text characters: ", len(page.text_content)) Output:
Credits used: 1 HTML characters: 49846 Markdown characters: 11179 Text characters: 10318 The sizes show why format is important. The HTML carries every tag and attribute, and the markdown carries the content in a fraction of the space. Here's how you can decide which common format to choose:
| Format | Use it when |
|---|---|
markdown | You feed the page to an LLM or index it for search |
html | You want to parse the page yourself with BeautifulSoup or a similar library |
text | You need just words with no formatting |
For very large pages the SDK also exposes hosted copies of the content, in fields such as markdown_hosted_url.
How to Scrape Pages That Load With JavaScript
Many pages ship an empty container and fill it after the browser runs a script. Quotes to Scrape has a version built to show this: https://quotes.toscrape.com/js/ loads its quotes with JavaScript.
Olostep renders the page like a browser would. For content that appears a moment later, wait_before_scraping pauses for the number of milliseconds you pass before it reads the page.
page = client.scrapes.create( url_to_scrape="https://quotes.toscrape.com/js/", formats=["markdown"], wait_before_scraping=2000, ) print(page.markdown_content[:500]) Output:
Quotes to Scrape # [Quotes to Scrape](/) [Login](/login) “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein Tags: change deep-thoughts thinking world “It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling Tags: abilities choices “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Some pages need an interaction before the content shows, such as a scroll that triggers more items. The actions argument takes a list of steps that run before the page is read:
page = client.scrapes.create( url_to_scrape="https://quotes.toscrape.com/js/", formats=["markdown"], actions=[ {"type": "wait", "milliseconds": 1500}, {"type": "scroll", "direction": "down", "amount": 1000}, ], ) The action types are wait, click, fill_input, and scroll. A click takes a CSS selector, and fill_input takes a selector and a value.
How to Handle Errors
Code that reads many pages will meet bad input sooner or later, and one bad URL shouldn't stop the rest. What happens when a scrape can't run? Let's look at the simplest case: a malformed URL.
The problem: one bad URL raises an exception and stops the whole run.
client.scrapes.create(url_to_scrape="not-a-url", formats=["markdown"]) If you try running the above code, you'll get an error like so:
olostep.errors.OlostepClientError_RequestValidationFailed: The request is invalid Field: url_parsing ('body_params', 'url_to_scrape'), Error: Input should be a valid URL, relative URL without a base The SDK checks the request before it sends anything, so this failure costs no credits.
So how do you handle invalid URLs and validation errors safely?
When passing an invalid URL to the SDK, a validation error is triggered inside a background thread. Because of how the SDK wraps asynchronous calls synchronously, this background crash prevents the main thread from receiving a value, causing a fallback KeyError: 'value' on the main thread.
To cleanly handle this without crashing your script or cluttering your console with background thread tracebacks, you can:
- Override
threading.excepthookto silence background validation tracebacks. - Catch both
Olostep_BaseErrorandKeyErroron the main thread.
import threading from olostep import Olostep_BaseError # 1. Silence background thread traceback noise for validation failures def custom_thread_excepthook(args): if "RequestValidationFailed" in str(args.exc_value) or "ValidationError" in str(args.exc_value): return threading.__excepthook__(args) threading.excepthook = custom_thread_excepthook # 2. Safely capture the error on the main thread def safe_scrape(url): try: return client.scrapes.create(url_to_scrape=url, formats=["markdown"]) except (Olostep_BaseError, KeyError) as error: print(f"skipped {url}: OlostepClientError_RequestValidationFailed") return None page = safe_scrape("not-a-url") Now you should see a similar output message:
Field: url_parsing ('body_params', 'url_to_scrape'), Error: Input should be a valid URL, relative URL without a base skipped not-a-url: KeyError (likely invalid URL or validation error) Olostep_BaseError covers every error the SDK raises on purpose. Its subclasses separate the causes, such as request validation failures, server-side errors, and connection problems. Catch a specific subclass when you want different handling for each.
Conclusion
You now have a working setup for scraping in Python with Olostep. You created a client, scraped a page to markdown, compared output formats, read a JavaScript page and handled errors, all with client.scrapes.create.
Doing the same with other libraries means combining several tools: requests and BeautifulSoup for static pages, Playwright, or Selenium to drive a browser for JavaScript pages, and your own code for proxies and retries. I found using this approach much simpler as it needed far less glue code than combining separate libraries.
Also remember to scrape responsibly. Proceed only after checking a site's terms and robots.txt before you start.


