Leonard Richardson

I feel similarly about Python. I never really had to interact with it beyond using it as a calculator on the command line until a project I was a part of earlier in the year had a utility Python script introduced and I needed to hack on it to make desired changes. Then, more recently, I started using BeautifulSoup[0] for personal scraping. Of course it feels more trivial now (and now I wish I documented my frustrations so I could be more specific and perhaps help out other Python learners in the future), but it does take a while to adjust to Pythonic code where everyone is using cultural conventions like _foo and __foo and you wonder what the hell __foo__ is[1].

More on-topic, searching is a major weakness of Python's docs. What's that? Python just got multi-line strings allowing embedded expressions? They're called f-strings? Let me go read about them[2]. I can't link it, but even using Startpage the immediate results are a post from RealPython and PEP498 and while both are great, they aren't terse enough to be frequently referenced. If you search for "formatted string literal"[3] in the docs it takes 10 results before you reach something explicitly talking about them. And then you need to follow the links a couple times to reach https://docs.python.org/3/reference/lexical_analysis.html#f-strings (notice it is anchored as "f-strings" which was the first term we searched for).

While the Python docs, when I've found them, have been adequate for me so far -- although I still…

I learned Ruby and Ruby on Rails much faster than I had any right to, when my new job in NYC required it. I practiced not by writing Ruby on the job, but writing Ruby to scrape Craigslist apartment listings and feeding them into a spreadsheet.

I started teaching myself computer programming in 2011/2012: I bought a bunch of Head First books published by O'Reilly, watched the entry-level CS video lectures that Stanford provided for free, participated in the first few Udacity courses, and I tried a couple Coursera classes. But I think I learned the most (in the shortest amount of time) when I went through your online book, 'The Bastards Book of Ruby'[0]

I loved the real-world examples you used: fetching Tweets through the Twitter API, web scraping with Nokogiri, and manipulating images with ImageMagick/RMagick are a few that stick out to me.

I'm sure you're doing great things at Stanford but I'm also confident that you could come out with a new book (covering the topics you spoke of above) that would help motivate people who are on the fence on whether or not they should start/continue learning computer programming.

I'd be happy to collaborate on something but I would argue that Python would be a better language to use than Ruby. Here are some of the topics I'd like to share with people (which I've found to be useful in my career/side-projects): reading data from CSV/Excel files (xlrd[1]), fetching data from APIs (requests[2]), web scraping (BeautifulSoup[3], Selenium[4]),…

Unlike Ruby Python doesn't have great facilities for writing making implicit APIs like this:

with Table():
header(...)
row(...)

You can do something like this but I'd recommend against it as it involves a lot of magic and has trade-offs which make it an uncommon choice. Instead you'd do something like:

with Table() as table:
table.header(...)
table.row(...)

This follows through to your template API as well. So you'd want to construct an object that represents your template, and have methods on that to add a table. For example:

page = sierra.Page(...)
with page.div(...), page.add_table() as table:
table.header(...)
table.row()

print(page) # __str__() should return the raw HTML response

This simplifies things, and you can use an in-memory StringIO object rather than a file. You can then return this string to flask as a text/html response and avoid Jinja2 entirely.

Also don't use CamelCase, please! It looks very unpythonic.

However you're really trying to create an API for producing a tree of nodes (some of which might logically map to individual tags, some of which might not) which are then serialized to HTML.

I'd recommend looking at how you can use BeautifulSoup for a lot of this. BS4 contains a lot of classes and functions for creating trees of HTML-like nodes, so you could use this rather than a StringIO object and directly writing "<div>" and having to support all of the attributes as method…

That really depends on your project and tech stack. If you're into Python and are going to deal with relatively static HTML, then the Python modules Scrapy [1], BeautifulSoup [2] and the whole Python data crunching ecosystem are at your disposal. There's lots of great posts about getting such a stack off the ground and using it in the wild [3]. It can get you pretty darn far, the architecture is solid and there are lots of services and plugins which probably do everything you need.

Here's where I hit the limit with that setup: dynamic websites. If you're looking at something like discourse-powered communities or similar, and don't feel a bit too lazy to dig into all the ways requests are expected to look, it's no fun anymore. Luckily, there's lots of js-goodness which can handle dynamic website, inject your javascript for convenience and more [4].

The recently published Headless Chrome [5] and puppeteer [6] (a Node API for it), are really promising for many kinds of tasks - scraping among them. You can get a first impression in this article [7]. The ecosystem does not seem to be as mature yet, but I think this will be foundation of the next go-to scraping tech stack.

If you want to try it yourself, I've written a brief intro [8] and published a simple dockerized development environment [9], so you can give it a go without cluttering your machine or find out what dependencies you need and how the libraries are called.

[1] https://scrapy.org/

[2]…

Under US law, between 1925 and 1964, it was necessary to renew copyright every 28 years.[1] There are many works which have fallen out of copyright for nonrenewal, and there are projects which are now going through copyright renewal records to determine which of these are in fact now public domain in the US. Roughly 3/4 of potentially copyrighted works (published since 1924 and initially registered) have proved to be public domain.

Once copyright has lapsed, it cannot be reinstated.

https://www.crummy.com/2019/07/22/0

https://www.nypl.org/blog/2019/05/31/us-copyright-history-1923-1964

________________________________

Notes:

1. Technically, the obligation existed until the 1976 act, at which time copyright status was automatic, but was retroactively waived for works never having lapsed, back to 1964, in the 1992 act. It's complicated. (NYPL link above.)

There are a few variations on CSV, but a great many variations on HTML. You could try to hack together some sort of quick and dirty regular expression based extractor, but if I were you I'd use http://www.crummy.com/software/BeautifulSoup/. It does a reasonably good job of parsing HTML, even invalid HTML such as is often found in the wild.

Then it should be pretty straightforward to scrape out the data you want. The scraper will be reasonably resilient and because the parsing code is separate from the code that deals with the structural details of the data source you're dealing with, it will be reasonably easy to maintain that when the provider redesigns their web pages.