Twitter makes it harder to read tweets

Yesterday I touched off a teacup-storm when I tweeted:

Twitter cut off the ability to read a tweet by fetching its URL with a normal HTTP GET. You need Javascript or an authenticated API call.

I know most people don't care. It's normal web browsing, with all the data-hoovering we've come to expect.

Just want to say that it matters.

--@zarfeblong, Dec 17th

Quite a few people started discussing this, and it wound up on this HackerNews thread.

What's going on here?

The problem here might not be clear. After all, if you click my tweet link in a regular web browser, you'll see the tweet and its replies, right?

"Twitter cut off the ability to read a tweet by fetching its URL with a normal HTTP GET. You need Javascript or an authenticated API call."

That's because a regular web browser supports Javascript, cookies, and everything else that's been thrown into the Web over the years. When you browse like this, Twitter can cookie-track you (notice that I'm using Safari's Private Browsing feature). They can nag you to log in. They can show you ads. All the usual stuff.

If you try turning off Javascript, you see this:

"JavaScript is not available."

(This is iCab, a third-party browser with some nice features. Twitter supports iCab just fine if JS is turned on.)

But let's reduce the situation to bare essentials: a raw HTTP GET request using curl.

curl -v https://twitter.com/zarfeblong/status/1339742840142872577

HTTP/2 400

This browser is no longer supported.

Please switch to a supported browser to continue using twitter.com. You can see a list of supported browsers in our Help Center.

I've elided most of the headers, but that's an HTTP "400: Bad Request" error response.

In the past, all of these requests would work; you could read a tweet this way. Twitter added warnings a while back ("Do you want the mobile site?") but you could get through if you wanted.

However, a few weeks ago, Twitter added a warning saying that on Dec 15th, they would remove support for legacy browsers. That's what changed this week. I didn't catch the exact time, but on Dec 15th, they started returning this error page which includes no information about the tweet.

What problems does this cause?

I admit up front that my browsing setup is weird. My default browser has JS turned off; it has a strict cookie expiration limit. This is hardcore privacy mode and most people don't do it. Lots of web pages work badly for me. But most of the web works at least a little bit -- when I go to a web page, I can usually see what's there.

(Of course, if I want to buy something or edit a Google doc, I need to switch modes. I use four different browsers and about six or seven different profiles with different levels of privacy protection. I told you it was weird.)

The point is, I am willing to trade off convenience for privacy. But as of Dec 15th, when I go to a Twitter URL, I see nothing. The tradeoff just got a little worse.

This isn't about me, though. From followup discussion:

• My friend Jmac is working on a tool which displays chat from several sources, including Twitter, in a single window. Twitter broke it.
• Keybase is an open platform which supports personal end-to-end encryption. It uses Twitter as one (of several) means of identity verification. Twitter broke that.
• Someone in the HN thread mentioned a script they use to check for broken links on their web site. Twitter broke that (for links to tweets).
• I've heard that previews of tweets on other chat/discussion platforms has also broken. (Not sure which ones, though.)

To be more ideological about it (yes I will), Twitter is asserting that people reading tweets is not their priority. Twitter is not about people communicating. They'll block communication if it pushes up their corporate numbers.

This was my summary from last night (slightly edited for clarity):

The point of the web is that a public URL refers to information. You can GET it. There may be all sorts of elaboration and metadata and extra services, but that HTTP GET always works.

Tools rely on this. Blogs pull previews via HTTP. RSS readers collect and collate posts. People peek pages using fast or low-bandwidth tools. It's not part of any company's value proposition or stock price; it just works. Until somebody breaks it.

The error message that appears now when I do this on Twitter (and the warning that popped up a few weeks ago) says that Twitter is dropping support for legacy browsers. But you support legacy browsers with an HTTP reply that contains the tweet text. That's all.

The only reason not to do this is that someone at Twitter said, "Why are we letting people just read tweets? That doesn't benefit us. They need The Full Twitter Experience." Which boils down to selling ads.

Yes, I could be using the API in various ways without directly hooking into the ad firehose. But then at least [Twitter] controls the experience -- [they] give out the API keys, [they] can decide to cut people off or otherwise control what's going on.

If I can just read a public tweet the same way every other web document works, then I can make the experience work how I want; I can have the conversation on my own terms and in my own framework.

To Twitter, this is anathema. So Twitter cut it off.

That's all. That's where we are. I don't expect different because Twitter is an ad company. I expect other changes in the future which will make Twitter even less hospitable to me. Maybe next week [or] maybe in ten years -- I'll find out when it happens.

--@zarfeblong, Dec 17th (or on nitter!)

Alternatives and workarounds

Of course there are many.

Twitter's public API

Twitter supports a web API. You can use it from any language you like (Perl, Python, whatever). In fact you can use it with curl or wget if you enjoy web plumbing.

As is usual for big web APIs, you have to register as an app developer and request an access key. This is free, but it puts the process under Twitter's control. In theory they can revoke your access if they think you're a baddie.

Also, the API has rate limits. This isn't a shock; every web service has some kind of rate limit, even if it's just Cloudflare or your ISP cutting you off. But Twitter has made it clear that they limit use of the API to make sure that third-party clients never get too popular. They don't exactly want to squash third-party clients, but they see them as a threat to be contained.

(See this 2012 blog post -- particularly the bit about "if you need a large amount of user tokens". I wrote more about this in 2018, when Twitter deprecated some functionality from their API.)

Nitter.net

Several people referred me to Nitter.net, an alternative Twitter front-end which doesn't require Javascript. Pretty much aimed at people like me! You can substitute Nitter in any public tweet URL and get a URL that works in all the ways we've been talking about. For example, https://nitter.net/zarfeblong/status/1339742840142872577:

curl -v https://nitter.net/zarfeblong/status/1339742840142872577

This downloads an HTML document which contains my original text, and all the replies. You can also visit https://nitter.net/zarfeblong to see my Twitter profile, except there's no ads, no promoted crap, and it loads fine (and fast!) without Javascript.

I'm sure that Nitter uses Twitter's API, mind you. It is, de facto, a third-party Twitter client. Which means that -- again, in theory -- Twitter could just cut it off. I don't know whether they're likely to. At any rate, at present, Nitter is a useful interface.

Faking your User-Agent

Turns out that if you're Google, you can GET tweets all you like! And pretending to be Google is pretty easy.

curl -v -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/bot.html)" https://twitter.com/zarfeblong/status/1339742840142872577

This sends Google's User-Agent string (instead of the standard curl string). Twitter happily sends back the same web page I was getting before Dec 15th.

Now, this web page asks me to turn on Javascript. But the text of my tweet is in there (along with people's replies in the thread). You can extract the text by looking for og:description, which is the standard web preview metadata.

Is this a viable workaround? It certainly works. But it's a hack. Twitter's policy is that I should not be allowed to get this page. Only Google can, because Google is king. I'm a peon. Their enforcement of this policy is crude and easy to cheat -- but that's a bug. Maybe they'll fix the bug. Maybe it's not important enough to fix. Again, I don't know.

It's also worth noting that the page you get this way is 586 kb. The Nitter.net equivalent is 28 kb. Twitter has piled a lot of junk into their UI.

So now what?

Honestly I haven't decided. I think I'm going to set up my default browser to redirect twitter.com URLs to nitter.net. That will take care of most of my day-to-day needs.

(I do use Twitter clients, by the way. Tweetbot on Mac, Echofon on iOS. But I try to use them in a mindful way. I don't want them to pop up just because I clicked somewhere.)

Mostly, this is an opportunity to look clearly at Twitter and remember that it does not love us. It's a commercial social network; we are its fuel and in its fire we shall burn. I use Twitter, but -- as with the web in general -- I want to remain aware of the tradeoffs.

For what it's worth, Mastodon never got traction for me. Sorry. My private social circles are now on Slack and Discord. Slack just got bought. Maybe it'll be okay. Discord's heel-turn is still in the unclear future. I don't know. We have to stay flexible.

So, you know... the usual story. Enjoy your holidays. Stay home and videochat. Play some games.

Reading Myst for detail

The VR remake of Myst is due out for Oculus Quest on the 10th. I will not be playing it this week because I'm not into VR or Facebook. The updated Myst will be released for other VR platforms and flatscreen PCs in 2021 -- I believe they said "early 2021" -- and I am happily anticipating that. The screenshots look amazing.

(Cyan has walked a painful line insisting that this is not a "Quest exclusive" release. It's just a game that releases on Quest this week, and other platforms at an unannounced future date. My personal theory is that Facebook walked into Cyan's office with a bucket of money and said "Make Myst VR a Quest exclusive!", and Cyan said "Oh, no, we really can't do that," and then there was a lot of pacing and gesticulating, and they wound up with this handwavy compromise where the PC version ships a couple of months later and nobody calls it a Quest exclusive. This is only a theory.)

Anyway, I thought it would be fun to go back and look at how little we know about the Myst story!

The original game does an incredible job of painting a complex story of betrayal with just a few brushstrokes of narration. But it's surprisingly hard to pin down exactly what happened. We know that Atrus trapped his sons in the red and blue books, and the boys trapped Atrus in D'ni... hang on, how did that work? Who was trapped first?

Here, try this puzzle. You've played Myst. Using only your knowledge of the original game, put these events in chronological order:

• (A) Atrus creates the red and blue trap books.
• (B) Atrus goes to D'ni and is trapped.
• (C) Atrus moves four linking books from the library to the protected places on Myst.
• (D) Atrus records a message to Catherine in the Forechamber imager.
• (E) Atrus tears pages from the red and blue books and scatters them.
• (F) Sirrus and Achenar burn the library on Myst.
• (G) Sirrus and Achenar do something ("used Catherine") to trick Atrus into going to D'ni.
• (H) Sirrus and Achenar enter the red and blue books and are trapped.
• (I) Sirrus and Achenar tear one page from Atrus's Myst linking book.

(I've put these in alphabetical order to avoid giving hints!)

---

Spoilers farther down the post. While you think about it, I'll mention some other recent Cyan news...

---

A month ago Cyan released a photo-sharing iPhone app called Crowbox. No, it's not a game; it really is a photo-sharing app. It uses peer-to-peer networking. You browse through your photos on your phone, select one, and everybody else standing near you can see it pop up on their phone. That's the whole gimmick. It's free, no ads, with a paid DLC option for extra features (more shareable albums).

Cyan teased this idea last year at Mysterium. Now they're pushing it as ideal for pandemic times -- you can show people your photo album while standing six feet away from them. (Outdoors, preferably, but you know this.)

They also posted an in-character Atrus journal touting the idea, in case you like your marketing ARG-style.

---

If you're into merch, Cyan has put new mugs, shirt, posters, and other stuff on their company store page. Some of this has art from the upcoming Myst; some spotlights fan artist Lauren Herda. I snagged a couple of mousepads, which are now sold out, ha ha.

The loss of Arecibo Observatory

We learned this morning that the Arecibo radio telescope has collapsed entirely. (Announcement; news story.) The hanging instrument platform broke loose and smashed into the dish below. No one was injured.

There are some preliminary photos of the wreckage on Twitter. I'm sure there will be more later. I won't link them. Here is a photo I took in March of 2019, when the observatory was fully operational:

Arecibo Observatory, March 2019

This is an ignominious end, but we knew it was coming. The first cable failed in August; more failed in November. On November 19th, engineers declared that the damage could not safely be repaired and the telescope would have to be decommissioned. We hoped for a controlled disassembly, but cables continued to unravel progressively, and this morning it all went.

So this hurts, but I did my grieving two weeks ago. Here is the twitter thread I wrote on the 19th:

Of all the terrible news this year, I am unexpectely wrecked by the decommissioning of the Arecibo radio telescope.

I saw the report of the first cable failure in August, but not the second, worse cable failure in November. It's bad and engineering assessments say it can't be safely repaired.

(Don't "what if" this. We believe that engineers know their shit.)

When I was in elementary school, I read Danny Dunn and the Voice from Space (Williams and Abrashkin). The book was written in 1967, a few years after Arecibo was completed. It was about a kid decoding messages from aliens. I loved everything about that.

The book's radio telescope was fictional ("Grendel Observatory" in England, thank you easily accessible ebooks) but it must have been inspired by the excitement around the brand-new world's-largest radio telescope in Puerto Rico.

I was so excited about SPAAACE that I started drawing a tiny radio telescope on things as my personal logo.

I did not wind up becoming an astronaut. But, for a while, I was "radiotelescope" on livejournal. I still have radiotelescope.dreamwidth.org, although I don't post there.

Hadean Lands source code book (a limited kickstarter reward) with radiotelescope logo

I still sometimes stick that radio telescope logo onto something, just for old time's sake. And SPAAAAACE.

Me visiting Arecibo

Here's me visiting that, early 2019. Like I said, one of the wonders of modern human civilization. We suck in a lot of ways but we build literal, monumental cathedrals to Science. Working Science.

The news stories say the observatory will still exist in some form, and maybe a new instrument can be built. I hope so. Every other headline this week [of Nov 19th] is about the absolute rage and contempt that's arisen against knowledge, science, and cooperative effort.

Sorry. This was supposed to be a positive thread. I want to be hopeful, but the future is hard these days and I'm scared for a lot more than astronomy.

Stop the fascists.

Apologies (again) for getting dark at the end. It's two weeks later now. The US fascist movement has lost its immediate hope of seizing power. (Don't ask me about 2022.) COVID vaccines are in production. We might still build a new telescope at Arecibo. It seems like a distant hope; we're still reaching towards the possibility of funding the Post Office. But we don't have to give up yet.

Civilization is a group effort. Stand in support of all of us.