1. Install Homebrew:

/bin/bash -c "$(curl -fsSL <https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh>)"

1. Install amazon’s command line tool:

brew install awscli parquet-cli

1. Configure amazon’s command line tool:

aws configure --profile neynar_parquet_exports

AWS Access Key ID [None]: the username from your 1Password entry
AWS Secret Access Key [None]: the password from your 1Password entry
Default region name [None]: us-east-1
Default output format [None]: json

1. Set this new profile to be the default (or you can use --profile ... on all of your aws commands):

export AWS_PROFILE=neynar_parquet_exports

1. List all the archive exports:

aws s3 ls s3://tf-premium-parquet/public-postgres/farcaster/v2/full/

You’ll see some output that will look something like this (the timestamps will likely be different):

2024-03-28 16:20:05          0 
2024-03-29 14:34:06 1877462159 farcaster-casts-0-1711678800.parquet
2024-03-29 14:39:11   21672633 farcaster-fids-0-1711678800.parquet
2024-03-29 14:40:07   15824832 farcaster-fnames-0-1711678800.parquet
2024-03-29 14:50:44 2823873376 farcaster-links-0-1711678800.parquet
2024-03-29 14:35:42 2851749377 farcaster-reactions-0-1711678800.parquet
2024-03-29 14:35:54   22202796 farcaster-signers-0-1711678800.parquet
2024-03-29 14:35:55   12937057 farcaster-storage-0-1711678800.parquet
2024-03-29 14:35:57   67192450 farcaster-user_data-0-1711678800.parquet
2024-03-29 14:35:59   72782965 farcaster-verifications-0-1711678800.parquet

The filename format is ${DATABASE}-${TABLE}-${START_TIME}-${END_TIME}.parquet. The timestamps bound the updated_at column.

You probably want to fetch the latest versions of each table the first time you build your database.

1. List all the incremental exports:

aws s3 ls s3://tf-premium-parquet/public-postgres/farcaster/v2/incremental/

2024-03-28 16:20:05          0 
2024-04-09 11:14:29    1011988 farcaster-casts-1712685900-1712686200.parquet
2024-04-09 11:14:25     200515 farcaster-fids-1712685900-1712686200.parquet
2024-04-09 11:14:25     231552 farcaster-fnames-1712685900-1712686200.parquet
2024-04-09 11:14:30     827338 farcaster-links-1712685900-1712686200.parquet
2024-03-29 14:35:42   51749377 farcaster-reactions-1712685900-1712686200.parquet
2024-04-09 11:14:26       8778 farcaster-signers-1712685900-1712686200.parquet
2024-04-09 11:14:26       6960 farcaster-storage-1712685900-1712686200.parquet
2024-04-09 11:14:30    1012332 farcaster-user_data-1712685900-1712686200.parquet
2024-04-09 11:14:30      10909 farcaster-verifications-1712685900-1712686200.parquet

1. List all the files for a specific time range:

aws s3 ls s3://tf-premium-parquet/public-postgres/farcaster/v2/incremental/ | grep "\\-1712685900\\-1712686200"