Contact

Data Engineer, Collection and Pipelines

You build the pipelines that collect data from public and licensed sources on a schedule and turn it into clean, versioned datasets. You record where each record came from and when it was captured, remove duplicates, resolve entities and deliver every version with its datasheet.

Apply

Team: Data, Kind: Full-time, Location: Remote, United States, Pay: $128,000 to $197,000 a year

Duties

  • Build collectors that gather records on a schedule from public sources, publishers, data holders and client systems.
  • Check each source's terms before anything is captured, and attach those terms to every record the source supplies.
  • Record the capture time of every record and keep the raw copy unchanged.
  • Keep each change in a source as a new version, with a log of what changed.
  • Design schemas for records and entities with the fields and relationships each client needs.
  • Remove exact and near duplicates, and filter out records outside scope by source, date, language or type.
  • Resolve one entity under different names, identifiers and units to one record.
  • Remove or replace personal details to the rules agreed with each client.
  • Deliver each version by authenticated API, scheduled export or agreed files, with its change log, quality results and datasheet.

What you bring

  • 3+ years in data engineering, building and running scheduled pipelines.
  • Strong Python and SQL.
  • Experience with schema design and data modeling.
  • Experience keeping collectors working when sources change their formats.
  • Care with data rights and personal data.

Nice to have

  • Orchestrators such as Airflow or Dagster.
  • PostgreSQL, ClickHouse and object storage.
  • Record linkage or entity resolution at scale.
  • Document parsing and OCR.

Pay and benefits

$128,000 to $197,000 a year

Full-time W-2 employment with health insurance, a retirement plan and paid leave.

No application deadline. We accept applications for this role on an ongoing basis.

Location

Remote, United States

Apply

Apply

We review applications as each team grows and contact you when there is a match.

Rillor is an equal opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status or any other status protected by law.

If you need an accommodation to apply, email careers@rillor.com.