Daniel Markow

​I have about 8 years of AWS under my belt. I still really don't like CloudWatch. The fact that someone at AWS thought it is a good idea to design a custom query language for logs is astonishing to me. And I know you can use SQL now or let AI generate the query for you - still it's not great. The UI is horrible. You get stuck with it however especially if budget is tight and custom solutions are out of the question. Or maybe you are using another AWS service like ECS Fargate to host your application. CloudWatch is deeply embedded in the AWS ecosystem. 

I do like DuckDB, though. It is a very capable file-based analytics database which just wants to plow through gigabytes of JSON logs on your laptop. Thus, I decided to make use of some of my free time and write my own command line tool cw-to-duck that exports CloudWatch logs directly into a DuckDB file. All you need to do is install AWS CLI, log in to your account with a role that has sufficient permissions to access your logs and run cw-to-duck. It also provides parsers for certain AWS services such that next to the logs_raw table the resulting DuckDB file will contain a handy logs_parsed table ready for analysis.