Repository files navigation

The social web translator. Fetches and converts data between social networks, HTML and JSON withmicroformats2,ActivityStreams/ActivityPub,Atom,JSON Feed, and more.

• About
• Using
  • Using the REST API
  • Using the library
• Troubleshooting/FAQ
• Future work
• Development
• Release instructions
• Related work
• Changelog

Granary is a library and REST API that fetches and converts between a wide variety of social data sources and formats:

• Facebook, Flickr, GitHub, Instagram, Mastodon, and Twitter native APIs
• Instagram and Facebook scraped HTML
• ActivityStreams 1.0 and 2.0 JSON, includingActivityPub
• HTML and JSON withmicroformats2
• Atom,RSS 2.0,JSON Feed
• Plain XML
• Bluesky/AT Protocol
• Nostr, with manyNIPs

Free yourself from silo APIchaff and expose the sweet social data foodstuff inside in standard formats and protocols!

Here's how to get started:

• Granary isavailable on PyPi. Install withpip install granary.
• Getting started docs.
• Reference docs.
• REST API and demo app atgranary.io.
• Source code on GitHub.

License: This project is placed in the public domain. You may also use it under theCC0 License.

The library and REST API are both based on theOpenSocial Activity Streams service. Let's start with an example. This code using the library:

fromgranaryimporttwitter...tw=twitter.Twitter(ACCESS_TOKEN_KEY,ACCESS_TOKEN_SECRET)tw.get_activities(group_id='@friends')

is equivalent to thisHTTP GET request:

https://granary.io/twitter/@me/@friends/@app/  ?access_token_key=ACCESS_TOKEN_KEY&access_token_secret=ACCESS_TOKEN_SECRET

They return the authenticated user's Twitter stream, ie tweets from the people they follow. Here's the JSON output:

{"itemsPerPage":10,"startIndex":0,"totalResults":12,"items": [{"verb":"post","id":"tag:twitter.com,2013:374272979578150912","url":"http://twitter.com/evanpro/status/374272979578150912","content":"Getting stuff for barbecue tomorrow. No ribs left! Got some nice tenderloin though. (@ Metro Plus Famille Lemay) http://t.co/b2PLgiLJwP","actor": {"username":"evanpro","displayName":"Evan Prodromou","description":"Prospector.","url":"http://twitter.com/evanpro",      },"object": {"tags": [{"url":"http://4sq.com/1cw5vf6","startIndex":113,"length":22,"objectType":"article"          },"..."],      },    },"..."]"..."}

The request parameters are the same for both, all optional:USER_ID is a source-specific id or@me for the authenticated user.GROUP_ID may be@all,@friends (currently identical to@all),@self,@search, or@blocks;APP_ID is currently ignored; best practice is to use@app as a placeholder.

Paging is supported via thestartIndex andcount parameters. They're self explanatory, and described in detail in theOpenSearch spec andOpenSocial spec.

When using theGROUP_ID@search (for platforms that support it — currently Twitter and Instagram), provide a search string via theq parameter. The API is loosely based on theOpenSearch spec, theOpenSocial Core Container spec, and theOpenSocial Core Gadget spec.

Output data isJSON Activity Streams 1.0 objects wrapped in theOpenSocial envelope, which puts the activities in the top-levelitems field as a list and adds theitemsPerPage,totalCount, etc. fields.

Most Facebook requests and all Twitter, Instagram, and Flickr requests will need OAuth access tokens. If you're using Python on Google App Engine,oauth-dropins is an easy way to add OAuth client flows for these sites. Otherwise, here are the sites' authentication docs:Facebook,Flickr,Instagram,Twitter.

If you get an access token and pass it along, it will be used to sign and authorize the underlying requests to the sources providers. See the demos on the REST APIendpoints above for examples.

Theendpoints above all serve theOpenSocial Activity Streams REST API. Request paths are of the form:

/USER_ID/GROUP_ID/APP_ID/ACTIVITY_ID?startIndex=...&count=...&format=FORMAT&access_token=...

All query parameters are optional.FORMAT may beas1 (the default),as2,atom,html,jsonfeed,mf2-json,rss, orxml (the default).atom supports a booleanreader query parameter for toggling rendering appropriate to feed readers, e.g. location is rendered in content whenreader=true (the default). The rest of the path elements and query params aredescribed above.

Errors are returned with the appropriate HTTP response code, e.g. 403 for Unauthorized, with details in the response body.

By default, responses are cached and reused for 10m without re-fetching the source data. (Instagram responses are cached for 60m.) You can prevent this by adding thecache=false query parameter to your request.

Include theshares=false query parameter to omit shares, eg Twitter retweets, from the results.

To use the REST API in an existing ActivityStreams/ActivityPub client, you'll need to hard-code exceptions for the domains you want to use e.g.facebook.com, and redirect HTTP requests to the correspondingendpoint above.

Facebook and Instagram are disabled in the REST API entirely, sadly.

See theexample above for a quick start guide.

Clone or download this repo into a directory namedgranary. Each source works the same way. Import the module for the source you want to use, then instantiate its class by passing the HTTP handler object. The handler should have arequest attribute for the current HTTP request.

The useful methods areget_activities() andget_actor(), which returns the current authenticated user (if any). See thefull reference docs for details. All return values are Python dicts of decoded ActivityStreams 1 JSON.

Themicroformats2.*_to_html() functions are also useful for rendering ActivityStreams 1 objects as nicely formatted HTML.

Check out theoauth-dropins Troubleshooting/FAQ section. It's pretty comprehensive and applies to this project too.

We'd love to add more sites! Off the top of my head,YouTube,Tumblr,WordPress.com,Sina Weibo,Qzone, andRenRen would be good candidates. If you're looking to get started, implementing a new site is a good place to start. It's pretty self contained and the existing sites are good examples to follow, but it's a decent amount of work, so you'll be familiar with the whole project by the end.

Pull requests are welcome! Feel free toping me in #indieweb-dev with any questions.

First, fork and clone this repo. Then, install theGoogle Cloud SDK and rungcloud components install cloud-firestore-emulator to install theFirestore emulator. Once you have them, set up your environment by running these commands in the repo root directory:

gcloud configset project granary-demopython3 -m venvlocalsource local/bin/activatepip install -r requirements.txt# needed to serve static files locallyln -s local/lib/python3*/site-packages/oauth_dropins/static oauth_dropins_static

Now, run the tests to check that everything is set up ok:

gcloud emulators firestore start --host-port=:8089 --database-mode=datastore-mode< /dev/null>& /dev/null&python3 -m unittest discover

Finally, run the web app locally withflask run:

GAE_ENV=localdev FLASK_ENV=development flask run -p 8080

Openlocalhost:8080 and you should see the granary home page!

If you want to work onoauth-dropins at the same time, install it in editable mode withpip install -e <path to oauth-dropins repo>. You'll also need to update theoauth_dropins_static symlink, which is needed for serving static file handlers locally:ln -sf <path-to-oauth-dropins-repo>/oauth_dropins/static oauth_dropins_static.

To deploy to production:

gcloud -q beta app deploy --no-cache granary-demo*.yaml

The docs are built withSphinx, includingapidoc,autodoc, andnapoleon. Configuration is indocs/conf.py To build them, first install Sphinx withpip install sphinx. (You may want to do this outside your virtualenv; if so, you'll need to reconfigure it to see system packages withvirtualenv --system-site-packages local.) Then, rundocs/build.sh.

Here's how to package, test, and ship a new release. (Note that this islargely duplicated in the oauth-dropins readme too.)

1. Run the unit tests.

   source local/bin/activate.cshCLOUDSDK_CORE_PROJECT=granary-demo gcloud emulators firestore start --host-port=:8089 --database-mode=datastore-mode< /dev/null>& /dev/null&sleep 5python -m unittest discoverkill %1deactivate

2. Bump the version number insetup.py anddocs/conf.py.git grep the old version number to make sure it only appears in the changelog. Change the current changelog entry inREADME.md for this new version fromunreleased to the current date.
3. Bump theoauth-dropins version specifier insetup.py to the most recent version.
4. Build the docs. If you added any new modules, add them to the appropriate file(s) indocs/source/. Then run./docs/build.sh. Check that the generated HTML looks fine by openingdocs/_build/html/index.html and looking around.
5. git commit -am 'release vX.Y'
6. Upload totest.pypi.org for testing.

   python setup.py clean build sdistsetenv ver X.Ysource local/bin/activate.cshtwine upload -r pypitest dist/granary-$ver.tar.gz

7. Install from test.pypi.org.

   cd /tmppython -m venvlocalsource local/bin/activate.cshpip uninstall granary# make sure we force Pip to use the uploaded versionpip install --upgrade pippip install mf2py==1.1.2pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple granary==$verdeactivate

8. Smoke test that the code trivially loads and runs.

   source local/bin/activate.cshpython# run test code belowdeactivate

   Test code to paste into the interpreter:

   importjsonfromgranaryimportgithubgithub.__file__# check that it's in the virtualenvg=github.GitHub('XXX')# insert a GitHub personal OAuth access tokena=g.get_activities()print(json.dumps(a,indent=2))fromgranaryimportatomprint(atom.activities_to_atom(a, {}))

9. Tag the release in git. In the tag message editor, delete the generated comments at bottom, leave the first line blank (to omit the release "title" in github), put### Notable changes on the second line, then copy and paste this version's changelog contents below it.

   git tag -a v$ver --cleanup=verbatimgit push&& git push --tags

10. Click here to draft a new release on GitHub. EntervX.Y in theTag version box. LeaveRelease title empty. Copy### Notable changes and the changelog contents into the description text box.
11. Upload topypi.org!

    twine upload dist/granary-$ver.tar.gz

12. Build the docs on Read the Docs: first chooselatest in the drop-down, then clickBuild Version.
13. On theVersions page, check that the new version is active, If it's not, activate it in theActivate a Version section.

Apache Streams is a similar project that translates between storage systems and database as well as social schemas. It's a Java library, and its design is heavily structured.Here's the list of formats it supports. It's mainly used byPeople Pattern.

Gnip similarlyconverts social network data to ActivityStreams and supportsmany more source networks. Unfortunately, it's commercial, there's no free trial or self-serve signup, andplans start at $500.

DataSift looks like broadly the same thing, except they offerself-serve, pay as you go billing, and they usetheir own proprietary output format instead of ActivityStreams. They're also aimed more at data mining as opposed to individual user access.

Cliqset's FeedProxy used to do this kind of format translation, but unfortunately it and Cliqset died.

Facebookused toofficiallysupport ActivityStreams, but that's also dead.

There are a number of products that download your social network data, normalize it, and let you query and visualize it.SocialSafe is one, although the SSL certificate is currently out of date.ThinkUp was an open source product, but shuttered on 18 July 2016. There's also the lifelogging/lifestream aggregator vein of projects that pull data from multiple source sites.Storytlr is a good example. It doesn't include Facebook, or Instagram, but does include a number of smaller source sites. There are lots of others, e.g. theLifestream WordPress plugin. Unfortunately, these are generally aimed at end users, not developers, and don't usually expose libraries or REST APIs.

On the open source side, there are many related projects.php-mf2-shim addsmicroformats2 to Facebook and Twitter's raw HTML.sockethub is a similar "polyglot" approach, but more focused on writing than reading.

Breaking changes:

• mastodon:
  • Switch AS1 actor and object ids fromtag: URIs to the real AS2 ids.

Non-breaking changes:

• bluesky:
  • Addpds_url kwarg toBluesky constructor.

• as2:
  • Add newset_content function to help keepcontent andcontentMap in sync.
  • to_as1: support integer secondsduration, which is non-standard but sent by some AP implementations,eg Funkwhale.
  • link_tags: addclass="hashtag" for hashtag (Tag,Hashtag) tags (bridgy-fed/#1634).
• bluesky:
  • Translate Blueskyapp.bsky.feed.post#tags to/from AS1tags (snarfed/bridgy-fed#1394).
  • Addauth kwarg toBluesky constructor to pass through as custom auth object torequests.get/post.
  • from_as1:
    • Bug fix for generating external embeds: convert HTMLcontent/summary to plain textdescription (bridgy-fed#1615).
    • Don't addapp.bsky.feed.post#tags that are overmaxGraphemes (64).
    • Add newraise_ kwarg to raiseValueError if a required object (eg the target of a like or repost) can't be fetched via ATProto.
    • Bug fix: ignoreinReplyTo for DMs.
    • Don't convert HTML links incontent with bad URLs to#link facets.
  • to_as1:
    • Bug fix: HTML-escape< and> characters, while preserving facet indices, so that they don't disappear (snarfed/bridgy-fed#1144).
  • preview/create:
    • Add support for follows.
  • to_external_embed: bug fix: handle compositeurl field.
• mastodon:
  • preview/create:
    • Add support for follows.
• nostr:
  • Add newNostr.delete method.
  • Add newsign function.
• source:
  • Source.postprocess_object: relax mention text matching withmentions=True, ignore server part of webfinger addresses.
  • Add newget_follows andget_followers methods, implement in Mastodon and Bluesky.

Breaking changes:

• as2:
  • from_as1: InLink objects (includingTags andMentions), converturl tohref. Before this, we left it asurl, which was incorrect AS2.

Non-breaking changes:

Standardize function and method names in all modules toto_as1,from_as, etc. Old method names are now deprecated but won't be removed until at least v9.0, if not later.

• as1:
  • Add newis_dm,recipient_if_dm,get_id, andis_audience functions.
• as2:
  • Addsensitive,indexable, anddiscoverable support.
  • Add newis_server_actor function (FEP-d556,discussion).
  • from_as1:
    • Always convert images to objects withtype: Image, never to bare string URLs (bridgy-fed#/1000).
    • Bug fixes for converting links to facets when the link text is the link URL.
  • to_as1:
    • Handle other types of tags better, eg non-standardHashtag and innertag field for name.
    • Bug fix for videos,mimeType goes in outer object, not instream.
    • Bug fix forto/cc with mixed dict and string elements.
  • link_tags: addclass="mention" forMention tags (bridgy-fed/#887).
• atom:
  • atom_to_activity/ies: Get URL fromlink for activities as well as objects. (Thanks @imax9000!)
• bluesky:
  • Translate Blueskyapp.bsky.feed.post#langs to/from AS1contentMap (which isn't officially part of AS1; we steal it from AS2).
  • Translate AS2sensitive on posts to Blueskygraphic-media self label, and many Bluesky self labels back tosensitive with content warning(s) insummary.
  • Translate AS1/2 DMs to/from Bluesky chats.
  • Translate video embeds in posts.
  • create/previewCreate:
    • IfinReplyTo isn't a Bluesky URL or AT URI, returnCreationResult instead of raisingValueError.
  • from_as1:
    • Convertarticles to external embeds with no post text.
    • Add newas_embed boolean kwarg to do the same thing for any object.
    • When truncating and adding a link to the original post, useid ifurl is not available (snarfed/bridgy-fed#1155).
    • If the input object hasinReplyTo orobject ortarget with no recognizable ATProto or Bluesky object, raiseValueError.
    • Omit images that aren't inblobs.
    • Bug fix for quote posts with text content that's longer than Bluesky's limit (snarfed/bridgy-fed#1197).
    • When aflag has multiple objects, use the first one that's an ATProto record.
    • Handle URLs more carefully, don't add link facets with invaliduris.
    • Populateblobs into external embedthumbs.
    • Parse image blobs and addaspectRatio to image record.
    • Bug fix: handle HTML links withtitle incontent correctly.
    • Bug fix: handle attachments with noid orurl.
  • to_as1:
    • Extract links fromapp.bsky.actor.profile#description and#summary intourl/urls fields.
    • Bug fix: first URL (singular) goes inurl, list of URLs goes inurls.
    • Bug fix: handle hashtags with regexp special characters.
    • Support string and bytes CIDs in blobrefs as well asCID instances.
    • Link hashtags to bsky.app hashtag search pages (bridgy-fed#1634).
  • Bluesky.get_activities: skip unknown record types instead of raisingValueError.
• microformats2:
  • object_to_json: Improve handling of items with multiple types by removinginReplyTo from likes, shares, etc (snarfed/bridgy-fed#941).
  • to_as1: don't crash on integer UNIX timestamps inpublished andupdated.
• rss:
  • Support image enclosures, both directions.
  • from_as1:
    • Bug fix: remove use of defaultauthor value'-' since RSS spec requires author values to include valid email addresses.
• source:
  • Source.postprocess_object: add newfirst_link_to_attachment boolean kwarg to fetch and generate a previewattachment for the first link in the HTMLcontent, if any.

Breaking changes:

• jsonfeed:
  • jsonfeed_to_activities: return AS1 objects, not activities.

Non-breaking changes:

• as1:
  • activity_changed: adddisplayName,summary fields.
  • is_public: returnFalse if the object/activity containsto that's empty or has only unknown aliases.
• as2:
  • Add support for theApplication,Block,Flag, andLink types.
  • Generalize actor logic into/from_as1 across all actor types, not justPerson.
  • Add newlink_tags function.
• atom:
  • activities_to_atom: handle image attachments withouturl field.
• bluesky:
  • to_as1:
    • Add support for:
      • app.bsky.embed.record
      • app.bsky.embed.recordWithMedia
      • app.bsky.feed.defs#notFoundPost
      • app.bsky.feed.generator
      • app.bsky.graph.block
      • app.bsky.graph.list
      • app.bsky.graph.listitem
      • com.atproto.admin.defs#repoRef
      • com.atproto.moderation.createReport#input
      • com.atproto.repo.strongRef
    • Add hashtag facet support.
    • Convert blobs in embeds togetBlob image URLs.
    • app.bsky.actor.profile: add HTML links for URLs insummary (snarfed/bridgy-fed#1065).
    • Escape HTML characters (<,>,&) inapp.bsky.actor.profiledescription field.
    • Bug fix forcreate/update activities with bare stringobject.
  • from_as1:
    • Add hashtag, mention, block, and flag support. Interprettags with missingobjectType as hashtags.
    • Guess missing indices in facets based on content text. Otherwise, if we still don't know a facet's indices, discard it.
    • Extract HTML links ( tags) from HTML content and convert to link facets (snarfed/bridgy-fed#976).
    • If an output string value is longer than itsmaxGraphemes ormaxLength in its lexicon, truncate it with an… ellipsis character at the end in order to fit. If this happens to post text, include a link embed pointing to the original post.
    • If the object has a video, include an external embed pointing to the original post and mark it as[Video] (snarfed/bridgy-fed#1078).
    • If the object has images, add the original post link to the end of the text, since Bluesky doesn't support both image and external embeds in the same post (bluesky-social/atproto#2575,snarfed/bridgy-fed#1106).
    • If anote hassummary - often used for content warnings in the fediverse - add it tocontent as a prefix instead of overridingcontent (snarfed/bridgy-fed#1001).
    • Populatereply.root properly in reply posts (snarfed/bridgy#1696).
    • Add neworiginal_fields_prefix kwarg to store original data in custom (off-Lexicon)*OriginalDescription and*OriginalUrl fields inapp.bsky.actor.profile and*OriginalText and*OriginalUrl fields inapp.bsky.feed.post (snarfed/bridgy-fed#1092).
    • Supportlexrpc.Client as well asBluesky forclient kwarg.
  • from_as1_to_strong_ref:
    • Addvalue boolean kwarg.
    • Changeclient kwarg fromBluesky tolexrpc.Client.
• microformats2:
  • Generalize actor logic across all actor types, not justperson.
  • json_to_object:
    • Strip leading# prefix (if present) from hashtagu-categorys.
    • Bug fix for whenname property is an object, eg anh-card.
  • object_to_json:
    • Convert bothid andurl insideinReplyTo toin-reply-to.
• nostr:
  • Handle connection closing while sending initial query.
• source:
  • Source.postprocess: when extracting @-mentions, defer to existing tag if it has the samedisplayName and hasurl.

• as1:
  • get_owner bug fix forpost,update,delete activities.
  • activity_changed: add newinReplyTo kwarg.
  • is_public: add newunlisted kwarg.
• as2:
  • to_as1: bug fix, preserveobjectType: featured for banner/header images even whenmediaType is also set.
  • is_public: add newunlisted kwarg.
  • from_as1:
    • Foricon field, prefer image types that areallowed by Mastodon.
    • Bug fix, handlestop-following with stringobject id.
• atom:
  • Add newextract_entries function.
  • activity_to_atom: default actor/author name to username.
  • atom_to_activities: support top-levelentry element as well asfeed.
  • atom_to_*:
    • addobject.author
    • defaultobjectType toarticle/note andverb topost
    • convertlink rel=self/alternate tourl
    • usedisplayName in objects instead oftitle
    • Interpret entrylink withoutrel as self link.
  • Ifentry.author doesn't have id or url, default them to feed author's.
• bluesky:
  • Implementcreate andpreview.
  • Fully support bothrecord andobject types infrom_as1 andto_as1. Useto_as1'stype kwarg andfrom_as1'sout_type kwarg to disambiguate.
  • ImplementBluesky.post_id.
  • Add newblob_to_url function.
  • Deleteas1_to_profile, switchfrom_as1 to return$type: app.bsky.actor.profile.
  • Convert HTMLsummary andcontent to plain text.
  • ImplementBluesky.user_to_actor,Bluesky.get_actor.
  • Don't log in (fetch an access token) eagerly in the constructor; wait until the client makes a call.
  • Prefer DID to handle in API calls that accept either.
  • at_uri_to_web_url: support lists.
  • web_url_to_at_uri: convert profile URLs likehttps://bsky.app/profile/snarfed.org to profile record URIs (at://snarfed.org/app.bsky.actor.profile/self) instead of repo URIs (at://snarfed.org).
  • Addfrom_as1_to_strong_ref.
  • Allow:s in record keys (atproto#2224).
  • to_as1:
    • Convert blobs,both new and old style, to PDSgetBlob URLs.
    • Add newuri kwarg.
    • Translatehandle tousername, add newrepo_handle kwarg.
    • Add support forapp.bsky.feed.repost,app.bsky.graph.defs#listView,app.bsky.feed.defs#blockedPost.
    • Addactor/author based onrepo_did.
    • Improveurl field: include custom handles, only userepo_did/handle forapp.bsky.actor.profile.
    • Handle bad facet indices that point inside Unicode code points (example;discussion).
    • Convert!no-unauthenticated label on profiles toAS1@unlisted audience target (bridgy-fed#828).
  • from_as1:
    • Addout_type kwarg to specify desired output type, egapp.bsky.actor.profile vsapp.bsky.actor.defs#profileViewBasic vsapp.bsky.actor.defs#profileView.
    • Addblobs kwarg to provide blob objects to use for image URLs.
    • Addclient kwarg to fetch and populate CIDs.
    • Handle mention tags pointing to bare DIDs.
    • Useparent asroot in replies. (Technically wrong in cases where the parent isn't the root, but we don't actually know the root. 🤷)
    • Bug fix: handle bare string URLs inimage field.
    • Bug fix: handle tags withouturl field.
    • Strip trailing slash from home page URLs in order to remove visible/ from rel-me verified links on Mastodon etc.
    • ConvertattributedTo to singular if it has only one element.
    • Ifname isn't set, fall back topreferredUsername or infer Webfinger handle fromid orurl.
    • Prioritize bsky.app profile URL before handle URL inurl field (bridgy#1640).
    • Convertbsky.appinReplyTo URLs toat:// URIs.
    • Tighten updatetime conversion to match theATProto recommended format.
• facebook:
  • RemoveFacebook.fql_stream_to_post.Facebook turned down FQL in 2016.
• github:
  • When converting data to AS1, usedisplayName in objects instead oftitle.
• mastodon:
  • get_activities bug fix: use query params for/api/v1/notifications API call, not JSON body.
  • Convert HTTP 200 responses witherror JSON field (eg from Sharkey) to 400/401 exceptions.
  • Prefermedia_attachments.remote_url when available since it may be more long-lived thanurl for remote statuses (bridgy#1675).
• microformats2:
  • object_to_json bug fix: handle singularinReplyTo.
  • json_to_object bug fix: handle list-valuedlocation.
• nostr:
  • get_*: return partial results when the websocket connection is closed prematurely.
  • to_as1: handle invalid NIP05 values (eg{})
• rss:
  • to_activities:
    • UseobjectType: note iftitle isn't set or is a prefix (possibly ellipsized) ofcontent/description.
    • Add support for images inmedia:content tags (#674).
• Source:
  • postprocess_activity/object: addmentions kwarg to convert @-mentions in HTML links tomention tags.

Highlights: Nostr, Blueskyget_activities, lots of improvements inas2 andmicroformats2, and more!

REST API breaking changes:

Twitter is dead, at least in the REST API.

Non-breaking changes:

• Add newnostr module!
• as1:
  • Addget_owner,targets.
  • Addaccept,reject,stop-following toVERBS_WITH_OBJECT and removerepost,it's not an AS1 verb.
  • Handleurl field list values (even though it's invalid AS1).
• as2:
  • to_as1:
    • ImproveVideo handling: supportLink objects inurl, extract stream URLs and types from linktags.
    • Coerce non-floatlatitude andlongitude to float, raiseValueError on failure.
    • Put image attachments intoimage as well asattachments (bridgy-fed#429).
    • Handle Hubzilla's composite object attachmentvalues.
    • Bug fix for nullmediaType inattachment andtags.
  • Add newTYPES_WITH_OBJECT constant.
  • Add newget_urls,address functions.
  • ImproveContent-Type compatibility withapplication/ld+json; profile="https://www.w3.org/ns/activitystreams".
  • Bug fix forUndo activities with bare string idobjects.
  • Revise HTML inPropertyValue attachments on actors to include full URL in anchro text to be compatible with Mastodon's profile link verification.
• atom:
  • activities_to_atom etc:
    • Switchcontent from XHTML to HTML inside CDATA to support non-XHTML input content (bridgy-fed#624.
    • Bug fix, handle bare string URLimage values.
    • Bug fix, emove incorrecttype="application/atom+xml" fromrel="self"link inentry.
    • RenderobjectType: comment attachments.
    • Remove invalid<a> element for tags.
    • Bug fix: avoid encoded< and> characters intitle (#629).
  • Bug fixes inactivity_to_atom/activities_to_atom for dict-valuedurl fields.
  • Render images in article/note attachments.
  • RenderobjectType: service attachments, eg Bluesky custom feeds.
• bluesky:
  • ImplementBluesky API class, includingget_activities.
  • Drop bundledapp.bsky/com.atproto lexicons, use lexrpc's instead.
  • Convert reposts, quotes, inline links, attached links, and mentions, both directions. Includes Bluesky facet (rich text) support.
  • Handle quote posts with attached images, both directions.
  • Handle likes, both directions.
  • Add newweb_url_to_at_uri function.
  • from_as1: handle link tags without start/end indices.
  • to_as1:
    • Add newtype kwarg.
    • Generate staging.bsky.app profile and post URLs.
    • Propagate profiledid into actorid.
    • Add unimplemented stub for custom feeds, egapp.bsky.feed.defs#generatorView.
  • Addas1_to_profile.
  • Bug fix for converting follows, both directions:subject inapp.bsky.graph.follow is followee, not follower. (That field is badly named!)
• jsonfeed:
  • activities_to_jsonfeed:
    • Bug fix, handle bare string values forimage andstream.
    • Bug fix: handle non-objectauthor.
• mastodon:
  • status_to_object: add/fix alt text handling for images.
• microformats2:
  • json_to_html:
    • HTML-escape tag and quote attachment names. FixesGHSA-4w4f-g49g-3f7j; thank you@janboddez!
  • json_to_object:
    • Improve handling of items with multiple types by usingpost type discovery more aggressively.
    • Normalize ISO-8601 format ofpublished andupdated timestamps.
  • object_to_json:
    • Bug fix, handle bare string URLimage values.
    • Normalize ISO-8601 format ofpublished andupdated timestamps.
    • Handle bare string ids forreplies andshares (usually from AS2.)
  • render_content:
    • Bug fix for bare stringauthor andactor values.
  • IncludeobjectType: service attachments, eg Bluesky custom feeds, in JSON and HTML output.
• rss:
  • from_activities: handle bare string idauthor.

Breaking changes:

• as2:
  • Interpret bare stringobject,inReplyTo, etc values as ids, convert them to bare strings orid instead ofurl.
• microformats2:
  • Convert simple stringin-reply-to,repost-of,like-of etc values to AS1 bare strings orids instead ofurls.

Non-breaking changes:

• Add newbluesky module forBluesky/AT Protocol!
• as1:
  • Add theorganization object type andACTOR_TYPES constant (based on AS2).
  • Add newget_ids,get_object, andget_objects functions.
• activity_changed: ignoreinReplyTo.author (snarfed/bridgy#1338)
• as2:
  • Support converting between AS1stop-following and AS2UndoFollow.
  • Support AS2Accept andReject for follows as well as event RSVPs.
  • Add support for theQuestion (ie poll),Organization, andDelete object types.
  • Convertto/cc to/from AS1to for public and unlisted.
  • Handletype: Document video attachments like Mastodon emits.
  • from_as1: bug fix for image objects withurl andvalue fields (for alt text).
  • from_as1: bug fix, handle bare string URLimage values.
  • from_as1: converturls.displayName toattachment.name (bridgy-fed#331).
  • from_as1: preserveinReplyTo object values as objects, inline single-element lists down down to just single element.
  • to_as1: useobjectType: featured for first image inimage field.
  • to_as1: populateactor intoobject.author forUpdates as well asCreates.
  • to_as1: convert Mastodon profile metadataPropertyValue attachments tourl composite objects withdisplayName.
  • Preserveto andcc values when converting both directions.
• atom:
  • Bug fix for rendering image attachments withoutimage field to Atom.
  • Bug fix forpublished andupdated in entries with objects, eg likes, reposts, RSVPs, bookmarks. Thanks@gregorlove! (#480)
  • Bug fix for contentactivity/ies_to_atom whenobject is present and empty.
  • Bug fix for objects with elements withoutobjectType in theto field.
• flickr:
  • get_activities: add support for thecount kwarg.
• github:
  • get_activities: add support for thecount kwarg.
• jsonfeed:
  • Switch fromwhite-space: pre CSS to converting newlines to<br>s because some feed readers follow it strictly and don't even line wrap (#456).
• mastodon:
  • Add compatibility support forTruth Social.
  • Handle truncated JSON API responses.
• microformats2:
  • json_to_object: drop backward compatibility support forlike andrepost properties.Background discussion.
  • json_to_object: add newrel_urls kwarg to allow attachingdisplayNames tourls based on HTML text ortitle attribute (bridgy-fed#331).
  • Add newjson_to_activities function.
  • hcard_to_html/maybe_linked_name: whenname is missing, use pretty URL as visible text.
  • Support theh-cardorg property.
  • json_to_object: handle compositersvp property value.
  • json_to_object: bug fix whenfetch_mf2 is True, handle when we run the authorship algorithm and fetch an author URL that has au-photo withalt.
• rss:
  • from_activities: fix item ordering to match input activities.

Breaking changes:

• Drop Python 3.6 support. Python 3.7 is now the minimum required version.
• Twitter, Instagram, Mastodon:
  • Dropget_activitiescache kwarg's support for App Engine memcache interface. It's now only used as a plaindict.get_activities will now make many small modifications, so if you pass an object that implements those as API calls, you'll probably want to batch those separately.
• Twitter, Mastodon, Flickr, GitHub:
  • create/preview: support the AS1favorite verb as well aslike. (bridgy#1345)
• Atom:
  • Switch to converting AS1id (instead ofurl) to Atomid.
• Reddit:
  • Implementget_actor.
• Mastodon:
  • create/preview: allow non-Mastodon replies, ie activities that includeinReplyTo URLs even if none of them point to a toot. (bridgy#1321)
  • Raiserequests.HTTPError withresponse.status_code 502 instead ofJSONDecodeError on non-JSON responses. This is synthetic, but more helpful for error handling.
• microformats2:
  • object_to_json and related functions: handle all escaped HTML entities, not just&<>.
  • Unifymicroformats2.prefix_image_urls andprefix_video_urls into a newas1.prefix_urls function.
• RSS:
  • Removeitunes:category. It has to beone of Apple's explicit categories, which we aren't prepared to validate, so don't try.
• ActivityStreams 2:
  • Translate bothurl andurls from AS1 into multi-valued AS2url field.
• Move a number of utility methods from theSource class to a newas1 module:object_type,merge_by_id,is_public,add_rsvps_to_event,get_rsvps_from_event,activity_changed,append_in_reply_to,actor_name,original_post_discovery.
• as1.original_post_discovery: remove deprecatedcache kwarg.

Non-breaking changes:

• ActivityStreams 2:
  • Fix spec compliance bug:icon andimage are singly valued, not multiply valued.
  • Add newis_public method andPUBLIC_AUDIENCE constant.
  • Prefer"objectType": "featured" first in theimage field when converting from AS1, last in theicon field. This matches the ActivityPub (Mastodon) convention of usingicon for profile pictures andimage for header images.
  • Propagateurl values into newPropertyValue attachments onPerson objects; these end up in Mastodon's "profile metadata" link fields.
  • to_as1: if an attachment'smediaType isimage/..., overrideobjectType and set it toimage.
• Twitter
  • Trim alt text in line between post preview and creation
  • Correctly trim Twitter alt text
• Facebook
  • Scraping: extract post id and owner id fromdata-ft attribute and_ft_ query param more often instead ofstory_fbid, which is now an opaque token that changes regularly. (facebook-atom#27)
• Instagram
  • Add newInstagram.scraped_json_to_activities method.
• GitHub
  • create andpreview: convert profile URLs to @-mentions, eghttps://github.com/snarfed to@snarfed (bridgy#1090).
    • get_activities withactivity_id now supportsfetch_replies andfetch_likes.
• Reddit
  • Addcache support toget_activities.
• REST API
  • Add new/scraped endpoint that acceptsPOST requests with silo HTML as input. Currently only supports Instagram. Requiressite=instagram,output=... (any supported output format), and HTML as either raw request body or MIME multipart encoded file in theinput parameter.
• microformats2
  • Add newextra andbody_class kwargs toactivities_to_html.
  • When convertingu-featured images to AS1, add new non-standard"objectType": "featured" field to distinguish them fromu-photo.
  • Convertp-note to AS1summary.
  • Bug fixes for convertingimage attachments tophoto.
• Source.original_post_discovery: add newmax_redirect_fetches keyword arg.

Breaking changes:

• Drop Python 3.5 support. Python 3.6 is now the minimum required version.

Non-breaking changes:

• RSS:
  • Add support for RSS input via newrss.to_activities function.
• Add newinclude_shares kwarg toget_activities, implemented for Twitter and Mastodon. Defaults toTrue. IfFalse, shares (retweets in Twitter, boosts in Mastodon) will be discarded and not returned. Also add a correspondingshares query param to the REST API.
• Instagram (scraping):
  • Handle media items with nouser object, add new fetch for comments.
  • AddInstagram.merge_scraped_comments().
• ActivityStreams 2:
  • Handle error whentype isn't a string.
• Reddit:
  • Implementget_activities() to fetch posts by the current user or a user specified withuser_id.
• Facebook scraping:
  • Skip "Suggested for you" posts.
  • Addlog_html kwarg toget_activities; defaults to False.
  • Miscellaneous bug fixes.
• JSONFeed:
  • Handle malformeditems.author element.

• Source.original_post_discovery: add newinclude_reserved_hosts kwarg, defaults toTrue.
• Instagram:
  • Update scraping to handle newfeed_v2 JSON format.
• Facebook:
  • Scraping: handle pictures, videos, link attachments, and text links in timeline/news feed posts.
• Mastodon:
  • Bug fix forget_activities() withfetch_mentions=True: handle notifications withstatus: null. Maybe happens when a status is deleted?
  • create/preview_create: support bookmarks. (Nothing special happens with them; theircontent is posted as a normal toot.)
• microformats2:
  • Stop renderingimage.displayName as visible text in HTML, since it's already in the<img>'salt attribute.
  • Addbookmark-of support.
  • Addprefix_image_urls() function.
  • Handle nullcontent in AS1/2 objects.
  • json_to_object bug fix for compositebookmark-of properties.
• Twitter:
  • create/preview:support large videos via async upload. We now passmedia_category=tweet_video to the chunked uploadINIT stage, and then make blockingSTATUS calls until the video is finished processing. (bridgy#1043)
  • create/preview: allow bookmarks. (bridgy#1045)
  • create/preview: allow non-Twitter replies, ie activities that includeinReplyTo URLs even if none of them point to a tweet. (bridgy#1063)
  • get_activities: support list ids as well as slugs.
  • Bug fixes for removing t.co links to quoted tweets.
  • Bug fix for multiple instances of the same link in tweet text.
  • get_activities(): raiseValueError on invaliduser_id.
• REST API: ported web framework from webapp2 to Flask. No user-visible behavior change expected.

• Add Python 3.8 support, drop 3.3 and 3.4. Python 3.5 is now the minimum required version.
• AddPixelfed! Heavily based on Mastodon.
• Standardize Instagram's and Facebook's scraping into new commonscraped_to_activities(),scraped_to_activity(), andmerge_scraped_reactions() methods.
• Atom:
  • Add thesummary element (#157).
• REST API:
  • Bug fix: URL-encode Unicode characters inLink HTTP headers (egrel=self,rel=header).
• Facebook:
  • Scraping now usesmbasic.facebook.com instead ofm.facebook.com.
• Flickr:
  • Add support for adding tags to existing photos (bridgy#857).
  • get_comment(): skip fetching comments from API ifactivity kwarg is provided and contains the requested comment.
• GitHub:
  • HandleHTTP 451 Unavailable for Legal Reasons responses (eg for DMCA takedowns) gracefully.
  • Add create/preview support for reactions on pull review request comments (ie URLs with#discussion_r... fragments).
• HTML/microformats2:
  • Addaria-hidden="true" to empty links (bridgy#947).
  • Bug fix: escape&,<, and> characters in bare mf2content properties (aaronpk/XRay#102).
  • json_to_object(): convertnickname tousername.
• JSON Feed:
  • Gracefully handle whencontent_html andcontent_text areincorrectly lists instead of strings.
• Instagram:
  • Include threaded (ie nested) comments in scraping (bridgy#958).
• Mastodon:
  • Bug fix for alt text with image attachments (bridgy#975).
  • Omit emptylimit paramfor compatibility with Pleroma (bridgy#977).
• Meetup:
  • create(): handle API errors and return the error message in theCreationResult (bridgy#921).
• Twitter:
  • Bug fix: URL-encode list names in API calls.
  • Bug fix: propagate alt text into AS1photo.displayName so that it gets all the way into microformats2 JSON and HTML (#183).
• Reddit:
  • Implementpost_id().
  • Cache user data fetched from the API for 5m to avoid repeating user profile API requests (bridgy#1021).when fetching multiple comments or posts from the same author
  • Bug fix: use 'displayName' instead of 'name' in AS1 objects for submissions.
  • Bug fix: use tag URIs for activity ids.
• ActivityStreams 2:
  • to_as1(): forCreate activities, include the activity actor's data in the object's author (snarfed/bridgy-fed#75).
  • to_as1(): convertpreferredUsername tousername.
  • from_as1(): convertusername topreferredUsername.
  • from_as1(): bug fix, makecontext kwarg actually work.

Breaking changes:

• Python 2 is no longer supported! Including theApp Engine Standard Python 2 runtime. On the plus side, thePython 3 runtime is now supported! See thislist of differences for more details.

Non-breaking changes:

• Migrate demo app and API to the App Engine Standard Python 3 runtime.
• Instagram:
  • Scraping: fetch 50 likes instead of 24. (snarfed/bridgy#898)
  • Scraping bug fix forget_actor() withuser_id.
• Twitter:
  • Addimage alt text support toget_activites() etc (#183).
• RSS:
  • Additunes:image,itunes:author, anditunes:category.
  • Strip HTML fromtitle element (#177).Background.
  • Always include author in items (#177).
  • Bug fix: extract feed image fromhfeed correctly.
  • Bug fix: don't crash onarticle ormention tags in items with enclosures.
• Atom:
  • Bug fix: extract feed image fromhfeed correctly.
• REST API:
  • Add HTTPHEAD support.
  • Add support for URL fragments withinput=html. If a fragment is provided, only that specific element is extracted and converted. (#185)
• GitHub:
  • Publish: preserve<code> tags instead of converting them to `s so that GitHub renders HTML entities like> inside them instead of leaving them escaped.Background.
• JSON Feed:
  • Handle malformed attachments better.
• microformats2:
  • Don't crash on stringcontext fields.
  • html_to_activities(): limit toh-entry,h-event, andh-cite items (#192).
• Thecache kwarg toSource.original_post_discovery() now has no effect.webutil.util.follow_redirects() has its own built in caching now.
• Added Meetup.com support for publishing RSVPs.

• Add Mastodon support!
• Add Python 3.7 support, and improve overall Python 3 compatibility.
• Update a number of dependencies.
• Switch from Python's built injson module toujson to speed up JSON parsing and encoding.
• Addduration andsize support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON.microformats2 support is still emerging for both. Both integer seconds andISO 8601 string durations are supported forduration. Integer bytes is used forsize everywhere. microformats2 HTML also includes human-readable strings, eg5.1 MB. (#169)
• Twitter:
  • [preview]_create(): detect attempts to uploadimages over 5MB and return an error.
• Facebook:
  • Addget_activities(scrape=True) for scraping HTML fromm.facebook.com. Requiresc_user andxs cookies from a logged in session (snarfed/bridgy#886).
  • Upgrade Graph API version from 2.10 to 4.0.
• Atom:
  • Bug fix for de-duping images in attachments.
• RSS:
  • Wrap all<description> element contents inCDATA sections.
  • Render images in<description> with HTML<img> tags (#175).
  • from_activities() bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)

• Convert AS2Mention tags to AS1objectTypemention (non-standard) and vice versa (snarfed/bridgy-fed#46).
• Twitter:
  • Bug fix for large block list fetches that get rate limited after a few successful requests.
  • Handle HTTP 403 + error code 200 when fetching retweets for a protected or otherwise unavailable tweet (bridgy#688).
  • Demote @-mentions fromperson-tags tomentions. Specifically, this means they'll no longer get rendered withu-category mf2.
• Instagram:
  • Disabled in the REST API entirely due to Instagram's aggressive rate limiting and blocking (bridgy#655).
  • Update scraping to handle replies in newedge_media_to_parent_comment field (#164).
  • Use cookie for all scraping HTTP requests, not just for likes.
• microformats2:
  • Revise whitespace handling; usewhite-space: pre CSS in HTML output.
• Facebook:
  • Bug fix: don't interpretphoto.php as username in post URLs.
• Atom:
  • Switch fromwhite-space: pre CSS back to converting newlines to<br>s because some feed readers (eg NewsBlur) follow it too strictly and don't even line wrap.
• RSS:
  • Default title to ellipsized content.

Breaking change: drop Google+ sinceit shuts down in March. Notably, this removes thegoogleplus module.

• Add RSS 2.0 output! (#124)
• All silos:
  • Switch users' primary URLs from web site to silo profile (#158).
• GitHub:
  • Don't enclose bare URLs in</> (snarfed/bridgy#850).
• Atom:
  • Bug fix for actors and attachments with multiple image URLs.
  • Bug fix for attachment author objects with no properties.
• Google+:
  • Drop from web UI and REST API sinceconsumer Google+ is shutting down entirely (more).
  • Switch from deprecated global API endpoint to G+ endpoint. Background insnarfed/bridgy#846,Google blog postand docs.
• Instagram:
  • Fix individual photo/video link urls for multi-photo/video posts.
  • Handleuser-provided alt text (#159).
• Twitter:
  • Update max video upload size from 5MB to 512MB (#162).
• /url: Return HTTP 400 when fetching the user's URL results in an infinite redirect.

Adddelete(). Currently includes Twitter and Flickr support.

• Instagram:
  • Make extra HTTP fetch (with cookie) to get individual likes (snarfed/bridgy#840).
  • Update scraping logic to handle feed HTML changes.
  • Link @-mentions in comments as well as photo/video captions.
• GitHub:
  • create/preview_create bug fixes for issues and comments on private repos.
  • Handle HTTP 410 Gone responses from REST API, eg when a repo has been deleted or issues for the repo disabled.
• Twitter:
  • Adddelete() andpreview_delete() for deleting tweets.
• Flickr:
  • Adddelete() andpreview_delete() for deleting photos.
• microformats2:
  • Addfollow-of support.
  • Only use quotation-of property for quote tweets, not URLs. (#155)
  • If a tag has startIndex/length, it gets linkified in the content, so don't also emit an mf2 child or HTML h-cite for it. (#155
• Atom:
  • Encode&s in author URL and email address too. (Thankssebsued!)
• AS2:
  • AddFollow support.

• Twitter:
  • Support ISO 8601 formatted created_at timestamps, which thearchive download uses, as well as RFC 2822 from the API.
  • create() andpreview_create(): support RSVPs. Tweet them as normal tweets with the RSVP content. (snarfed/bridgy#818)
  • create() andpreview_create(): support alt text for images, via AS1displayName. (snarfed/bridgy#756).
• Instagram:
  • Add global rate limiting lock for scraping. If a scraping HTTP request gets a 429 or 503 response, we refuse to make more requests for 5m, and instead short circuit and return the same error. This can be overridden with a newignore_rate_limit kwarg toget_activities().
• GitHub:
  • Addtag support tocreate/preview_create to add label(s) to existing issues (snarfed/bridgy#811).
  • Escape HTML characters (<,>, and&) in content increate() andpreview_create() (snarfed/bridgy#810).
  • get_activities() andget_comment() now returnValueError instead ofAssertionError on malformedactivity_id andcomment_id args, respectively.
  • get_activities() bug fix for issues/PRs with no body text.
  • Switch from GraphQL to REST API for creating comments and reactions, since GraphQL hits authorization errors on many org repos. (snarfed/bridgy#824)
  • Improve GraphQL support for comments and users.
• Atom:
  • Shorten and ellipsize feed title when necessary (#144).
• microformats2:
  • Upgrade mf2py to improve a few things likeimplied p-name detection and whitespace handling (#142, fixes#145,snarfed/bridgy#756,snarfed/bridgy#828).
  • Supportalt attribute in<img> tags (snarfed/bridgy#756).

• Add Python 3 support! Granary now requires either Python 2.7+ or Python 3.3+.
• Instagram:
  • Fix scraping profile pages.
• Twitter:
  • Update character counting to handle Twitter change that now auto-linksall ccTLDs.Background.
• GitHub:
  • Bug fix forget_activities() with deleted issues and repos.
• microformats2:
  • object_to_json(): convert tags to simple strings in thecategory property, not full nested objects likeh-cards (#141).
  • Special case GitHub issues that are in-reply-to a repo or its/issues URL to be objectTypeissue.
  • Render simple string categories in HTML output.

This release is intentionally small and limited in scope to contain any impact of the Python 3 migration. Itshould be a noop for existing Python 2 users, and we've tested thoroughly, but I'm sure there are still bugs. Please file issues if you notice anything broken!

• Add GitHub!
  • get_activities() supports issues and pull requests, including comments and reactions. It's currently based on notifications, so it's best effort, not comprehensive, and only includes recently active issues/PRs.
  • create() andpreview_create() support issues, comments,stars, andreactions.
• Twitter:
  • Prefer MP4 and other video/... content types to HLS (.m3u8) etc.Background.
  • Prefer HTTPS URLs for media images.
  • get_activities(): Support @-prefixed usernames inuser_id.
• Facebook:
  • Support newrecurring aka multi-instance events.create() andpreview_create() now only support RSVPs to individual instances of multi-instance events, to match the Facebook API itself.
  • Try harder to find original (full) sized photo URLs, specifically_o.jpg files instead of_s.jpg.
  • create() bug fix for photo and image URLs with unicode characters.
  • Fixed bug whereget_activities(user_id=...) included the authenticated user's own recent photos, albums, and news publishes.
• Instagram:
  • Extract more user (author) data from scraped profile pages.
  • Fix home page feed scraping.
• microformats2, Atom:
  • Add enclosures for image attachments.
  • Bug fixes for rendering image, video, and audio attachments inside shares and attachments. De-dupe images.
• microformats2:
  • Handle simple string-only author properties.
  • Addfetch_mf2 kwarg tojson_to_object() for fetching additional pages over HTTP to determine authorship.
  • Generate explicit blankp-name in HTML to prevent old flawedimplied p-name handling (#131).
  • Fixshare verb handling inactivity_to_json() andactivities_to_html() (#134).
  • Remember which content contains HTML, preserve newlines in it, and don't translate those newlines to<br>s (#130).
• Atom:
  • Fix timezone bugs inupdated andpublished.
• JSON Feed:
  • Omit title from items if it's the same as the content. (Often caused by microformats2's impliedp-name logic.)

• Moved web site and REST API togranary.io!granary-demo.appspot.com now 301 redirects.
• Twitter:
  • Update the publish character limit to 280.Background.
  • Fix abug in preview_create that auto-linked @-mentions inside URLs, e.g. Medium posts.
  • Support videos and animated GIFs inget_activities() etc.
• Instagram:
  • Add cookie query param to REST API to allow scraping that logged in user's feed.
• HTML (including Atom content):
  • Render image, video, and audio attachments more often and consistently.
  • Include microformats2u-photo,u-video, andu-audio classes more often and consistently.
• Atom:
  • Addatom_to_activities() for converting full feed documents.
  • Add to REST API and web UI.
  • Include source URL inrel=alternate link as well as actor/author URL (#151).
• JSON Feed:
  • Fix bug that omitted title in some cases (#122).

• AddActivityStreams 2.0! Newas2 module includesto_as1() andfrom_as1() functions. Currently supported: articles, notes, replies, likes, reposts, events, RSVPs, tags, attachments.
• Atom:
  • Add newatom_to_activity() function for converting Atom to AS1.
  • Add email field to author, if provided.
• JSON Feed:
  • Raise ValueError on bad (non-dict) input.
• REST API:
  • Addas2 value forformat andinput. Revise existing ActivityStreams and microformats2 value names toas1,as1-xml, andmf2-json. Old valuesactivitystreams,json,json-mf2, andxml are still accepted, but deprecated.

• AddJSON Feed support to both library and REST API.
• Twitter:
  • Addget_blocklist().
  • Bug fix for creating replies, favorites, or retweets of video URLs, e.g.https://twitter.com/name/status/123/video/1 .
  • Bug fix for parsing favorites HTML to handle a small change on Twitter's side.
  • post_id() now validates ids more strictly before returning them.
• Facebook:
  • Improve heuristic for determining privacy of wall posts from other users.
  • Support GIFs in comments (attachment typesanimated_image_autoplay andanimated_image_share).
  • Upgrade Graph API fromv2.6 tov2.10.
• Instagram:
  • Update scraping to handle new home page (ie news feed) JSON schema, which changed sometime around 2017-02-27. (Profile pages and individual photo/video permalinks still haven't changed yet.)
• microformats2:
  • Addu-featured to ActivityStreamsimage.
  • Improveh-event support.
  • Minor whitespace change (added

    ) when rendering locations as HTML.

  • post_id() now validates ids more strictly before returning them.
  • Fix bugs in converting latitude and longitude between ActivityStreams and mf2.
• Google+:
  • Update HTML scraping to handle changed serialized JSON data format.
• Atom:
  • Add newactivity_to_atom() function that renders a single top-level<entry> instead of<feed>.
  • Add newreader query param for toggling rendering decisions that are specific to feed readers. Right now, just affects location: it's rendered in the content whenreader=true (the default), omitted whenreader=false.
  • Include author name when rendering attached articles and notes (e.g. quote tweets).
  • Only include ASactivity:object-type andactivity:verb elements when they have values.
  • Render AS image and mf2 u-photo if they're not already in content.
  • Renderthr:in-reply-to fromobject.inReplyTo as well asactivity.context.inReplyTo.
• REST API:
  • Fix bugs in html => json-mf2 and html => html conversions.
• Upgrade brevity to 0.2.14 for a couplebugfixes.

• microformats2:
  • Interpret h-cite andu-quotation-of (experimental) as attachments, e.g. for quote tweets.
  • Convertaudio andvideo properties to AS attachments.
• Twitter:
  • Linkify @-mentions and hashtags inpreview_create().
  • Support creating quote tweets from attachments with Twitter URLs.
  • When converting quote tweets to AS, strip quoted tweet URL from end of text.
  • Raise ValueError whenget_activities() is passedgroup_id='@search' but notsearch_query.
• Instagram:
  • Improve HTML scraping error handling.
  • Supportmulti-photo/video posts.
• Facebook:
  • Disable creating "interested" RSVPs, since Facebook's API doesn't allow it.
• Atom:
  • Supportmedia enclosures for audio and video attachments.
• Source.get_activities(): start raising ValueError on bad argument values, notably invalid Facebook and Twitter ids and Instagram search queries.
• Fix rendering and linkifying content with Unicode high code points (ie above the 16-bit Basic Multilingual Plane), including some emoji, on "narrow" builds of Python 2 with--enable-unicode=ucs2, which is the default on Mac OS X, Windows, and older *nix.

• Twitter:
  • Handle new "extended" tweets with hidden reply-to @-mentions and trailingURLs for media, quote tweets, etc. Background:https://dev.twitter.com/overview/api/upcoming-changes-to-tweets
  • Bug fix: ensure like.author.displayName is a plain unicode string so that itcan be pickled normally, e.g. by App Engine's memcache.
  • Bug fix: handle names with emoji correctly in favorites_html_to_likes().
  • Bug fix: handle search queries with unicode characters.
• Atom:
  • Render full original quoted tweet in retweets of quote tweets.
• microformats2 HTML:
  • Optionally follow and fetch rel="author" links.
  • Improve mapping between microformats2 and ActivityStreams 'photo' types. (mf2 'photo' type is a note or articlewith a photo, but AS 'photo' typeis a photo. So, map mf2 photos to underlying type without photo.)
  • Support location properties beyond h-card, e.g. h-adr, h-geo, u-geo, and evenwhen properties like latitude and longitude appear at the top level.
• Error handling: return HTTP 502 for non-JSON API responses, 504 for connection failures.

• REST API:
  • Support tag URI for user id, app id, and activity id.
• Twitter:
  • Better error message when uploading a photo with an unsupported type.
  • Only include original quote tweets, not retweets of them.
  • Skip fetching retweets for protected accounts since the API call always 403s.
• Flickr:
  • Better username detection. Flickr's API is very inconsistent about usernamevs real name vs path alias. This specifically detects when a user name isprobably actually a real name because it has a space.
  • Uploading: detect and handle App Engine's 10MB HTTP request limit.
  • Bug fix in create: handle unicode characters in photo/video description,hashtags, and comment text.
• Atom:
  • Bug fix: escape &s in attachments' text (e.g. quote tweets).
  • Bug fix: handle multiply valued 'object' fields in ActivityStreams 1 activities.
• GitHub:
  • Switch creating comments and reactions from GraphQL to REST API (bridgy#824.

• Bump oauth-dropins requirement to 1.4.

• REST API:
  • Cache silo requests for 5m by default, 60m for Instagram because they aggressively blocking scraping. You can skip the cache with the new cache=false query param.
• Facebook:
  • Upgrade from API v2.2 to v2.6.https://developers.facebook.com/docs/apps/changelog
  • Add reaction support.
  • De-dupe event RSVPs by user.
• Twitter:
  • Switch create() to use brevity for counting characters.https://github.com/kylewm/brevity
  • Fix bug in create() that occasionally incorrectly escaped ., +, and - characters.
  • Fix text rendering bug when there are multipl photos/videos.
  • When replying to yourself, don't add a self @-mention.
• Instagram:
  • Fix bugs in scraping.
• Upgrade to requests 2.10.0 and requests-toolbelt 0.60, which support App Engine.

• Updateoauth-dropins dependency to >=1.3.

• Support posting videos! Currently in Facebook, Flickr, and Twitter.
• Instagram:
  • Add support for scraping, since they'relocking down their API and requiring manual approval.
  • Linkify @-mentions in photo captions.
• Facebook:
  • FetchOpen Graph stories akanews.publish actions.
  • Many bug fixes for photo posts: better privacy detection, fix bug that attached comments to wrong posts.
• Twitter:
  • Handle all photos/videos attached to a tweet, not just the first.
  • Stop fetching replies to @-mentions.
• Atom:
  • Render attachments.
  • Addxml:base.
• microformats2:
  • Load and convert h-card.
  • Implement full post type discovery algorithm, using mf2util.https://indiewebcamp.com/post-type-discovery
  • Drop support for h-as-* classes, both incoming and outgoing. They're deprecated in favor of post type discovery.
  • Drop old deprecatedu-like andu-repost properties.
• Misc bug fixes.
• Set up Coveralls.

• Improve original post discovery algorithm. (bridgy #51)
• Flickr tweaks. (bridgy #466)
• Add mf2, activitystreams, atom, and search to interactive UI. (#31,#29)
• Improved post type discovery (using mf2util).
• Extract user web site links from all fields in profile (e.g. description/bio).
• Add fabricated fragments to comment/like permalinks (e.g. #liked-by-user123) so that object urls are always unique (multiple silos).
• Improve formatting/whitespace support in create/preview (multiple silos).
• Google+:
  • Add search.
• Facebook:
  • Fetch more things in get_activities: photos, events, RSVPs.
  • Support person tags in create/preview.
  • Prevent facebook from automatically consolidating photo posts by uploading photos to "Timeline Photos" album.
  • Include title in create/preview.
  • Improve object id parsing/resolving.
  • Improve tag handling.
  • Bug fix for fetching nested comments.
  • Misc improvements, API error/flakiness handling.
• Flickr:
  • Create/preview support for photos, comments, favorites, tags, person tags, location.
• Twitter:
  • Create/preview support for location, multiple photos.
  • Fetch quote tweets.
  • Fetching user mentions improvements, bug fixes.
  • Fix embeds.
  • Misc AS conversion improvements.
• microformats2:
  • Improve like and repost rendering.
• Misc bug fixes.
• Set up CircleCI.

• Add Flickr.
• Facebook:
  • Fetch multiple id formats, e.g. with and without USERID_ prefix.
  • Support threaded comments.
  • Switch from /posts API endpoint to /feed.
• Google+:
  • Support converting plus.google.com HTML to ActivityStreams.
• Instagram:
  • Support location.
• Improve original post discovery algorithm.
• New logo.

• Bug fix for atom template rendering.
• Facebook, Instagram: support access_token parameter.

• Initial PyPi release.