Apache Beam - How to control/limit the concurrency process of ParDo ? (programming.dev)

submitted 6 months ago by hahattpro@programming.dev to c/ask_experienced_devs@programming.dev

0 comments fedilink hide all child comments

The problem is I am using beam.ParDo to call api, to enrich my data.

pipeline | "read_csv" >> beam.io.ReadFromCsv("...") 
                | "get_info_api" >> beam.ParDo(GetApiDoFN()) 
                | "write_output" >>  beam.io.WriteToBigQuery(...)

The problem is that api have limit. I need a way to control so it do not overload api or get 429 error.

I would like to set api call should not exceed 4 call / second.

How should I do it ?

no comments (yet)

sorted by: hot top controversial new old

there doesn't seem to be anything here

this post was submitted on 19 May 2024

15 points (100.0% liked)

Ask Experienced Devs

1232 readers

1 users here now

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 1 year ago

MODERATORS

Ategon@programming.dev