Monitoring your ECS Fargate logs with AWS FireLens and LOGIQ

7 min readSep 15, 2021

Amazon Web Services (AWS) Fargate is a serverless compute engine for containers that works on Amazon ECS and EKS. AWS FireLens is a log routing agent built for ECS containers and enables administrators to use task definition attributes to route logs to external log aggregators. AWS FireLens unifies data collection across the ECS cluster, and its pluggable architecture allows adding data sources, parsers, filters/buffering, and output plugins.

In a previous article, we took you through how AWS FireLens works and configuring AWS FireLens to forward AWS Fargate logs for unification and analysis on LOGIQ. While that guide still holds good for users of AWS FireLens versions 1.3 and below, this guide takes you through how to forward container logs for Fargate 1.4 and above to LOGIQ using AWS FireLens.

But first, for the uninitiated, let us tell you why you should use LOGIQ as your only log data convergence, management, and analytics platform.

What is LOGIQ?

LOGIQ is a unified data platform for monitoring, observability, log aggregation, and analytics with an infinite storage scale to bring simple and powerful logging to the masses. It uses object storage for data at rest and enables you to centralize logging for your entire stack — from applications and APIs to traditional and serverless infrastructure components. The use of object storage also means that neither do we dictate how much log data you can store and for how long, nor do we force you to favor logging specific components of your environment over others — you get to log everything.

The LOGIQ platform includes:

A User Interface (UI)
A command-line tool
Native integrations with widespread infrastructure services
A monitoring stack for time-series metrics
A log analytics stack for log data, and;
A free library for pre-built, no-code monitoring and observability dashboards

There are plenty of deployment options available for those who haven’t installed LOGIQ yet. With LOGIQ, you can take either the SaaS route or the PaaS route. For a cloud instance of LOGIQ SaaS, head over to our website and register for a 14-day free trial of LOGIQ SaaS. We’ll have your instance set up in no time.

On the other hand, there are plenty of deployment options to choose from if you’d like to try out LOGIQ PaaS. You could go for the free-forever LOGIQ PaaS Community Edition or try out our straightforward deployments on Kubernetes, AWS, MicroK8s, or Docker.

If you’d like to get up and running quickly, try out our Ansible playbook for installing LOGIQ on Ubuntu using MicroK8s.

Configuring AWS FireLens to forward logs

Possibly the most significant advantage with AWS FireLens is that you can connect it to almost any service endpoint as long as data sinks can process general-purpose JSON over HTTP, FluentForward, or TCP protocols. FireLens magically transforms log outputs of ECS containers and generates the routing configuration needed for sending logs to the logging service.

The “awsfirelens” log driver allows you to specify Fluentd or Fluent Bit output plugin configurations. Your application container logs are routed to a sidecar or independent Firelens container inside your cluster, which further routes your container logs to their destination as defined in your task “logConfiguration“. Additionally, you can use the options field of the `FireLensConfiguration` object in the task definition to serve any advanced use case.

"firelensConfiguration" : {
     "type" : "fluentbit",
     "essential":true,
     "options" : {
        "config-file-value" : "arn:aws:s3:::mybucket/myFile.conf",
        "config-file-type" : "s3"
     }
  }

The configuration above holds good for anyone using Fargate 1.3 or lower. For those on Fargate 1.4 or above, we’ve taken the AWS-provided Fluent Bit image and added the LOGIQ cluster-id configuration in firelens.conf so that it’s packaged inside the AWS Fluent Bit image. Tasks hosted on Fargate only support the file configuration file type. So, instead of specifying it as s3, we’ll use the AWS Fluent Bit image packaged with our custom configuration.

"firelensConfiguration": {
   "type": "fluentbit",
   "options": {
       "config-file-type": "file",
       "config-file-value": "/firelens.conf"
   }
}

Creating a custom AWS Fluent Bit image

To package the LOGIQ custom configuration with the AWS Fluent Bit image, do the following.

Clone the following repository and navigate to the custom-fluent-bit-image folder: https://github.com/logiqai/logiq-firelens-fluentbit
Run the following command to build a new docker image with the fluent-logiq configuration:
docker build -t logiq-config
This image should now be pushed into the private Docker registry for us to be able to use it in the container later on. In order to push the image to the AWS Private registry, tag the image by running the following command: docker tag image-id Username.dkr.region.amazon.com/logiqconfiguration

Doing this should match the repository on AWS ECR in order to push the image successfully. For more information, read: https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html).

Run the following command to get the Docker login credentials to AWS ECR. Use the same credentials to login: aws ecr get-login
Once logged in, push the image to the private ECR registry using the command: docker push Username.dkr.region.amazon.com/logiqconfiguration
Reference the configuration file path in the FireLens configuration, as shown below.

"firelensConfiguration": {
   "type": "fluentbit",
   "options": {
       "config-file-type": "file",
       "config-file-value": "/firelens.conf"
   }
}

Configuring AWS FireLens to forward your container logs

There are two ways to configure log forwarding from your ECS Fargate containers to LOGIQ. You could either configure log forwarding globally for an entire Fargate cluster, or you can access and configure each individual container. We recommend configuring log forwarding globally on the cluster unless your use case specifically requires individual container configurations.

To configure log forwarding on your Fargate cluster globally, do the following.

Access your Fargate cluster from your AWS Console.
Scroll down to Log Router Integration
Select Enable FireLens integration.
Select fluentbit from the Type dropdown menu.
Provide the path to your custom AWS Fluent Bit image that contains the LOGIQ configuration in the Image field.
Click Apply.

Next, click Configure via JSON.
Edit and paste the following logConfiguration.

"logConfiguration": {
                "logDriver": "awsfirelens",
                "secretOptions": null,
                "options": {
                    "tls.verify": "off",
                    "net.keepalive": "off",
                    "Format": "json",
                    "Header Authorization Bearer ": "<Logiq token>",
                    "compress": "gzip",
                    "Port": "443",
                    "Host": "your Logiq hostname",
                    "tls": "on",
                    "URI": "/v1/json_batch",
                    "Name": "http"
			}
		}

Your ECS Fargate logs are now configured to be sent to LOGIQ using AWS FireLens.

Another way to configure log forwarding using AWS FireLens to LOGIQ is to configure it on each container. In this method, you will access each container definition and configure log forwarding for each container.

To set up log forwarding on an ECS container, do the following.

Access a container definition and scroll down to the Log configuration section.
From the Log driver dropdown menu, select awsfirelens.
Under Log options, update the log configuration options to reflect the configuration we’ve set for LOGIQ.

Click Apply.
Repeat these steps for all the applicable containers in your Fargate cluster.

Your container logs are now configured to be forwarded to LOGIQ using AWS FireLens.

After completing either of the above configuration steps, you’ll see a new container named log_router figuring in your list of containers for the cluster.

Since containers expect the CLUSTER_ID as a parameter, let’s edit the environment variable on the log_router container and add a name to identify your cluster, as shown in the following image.

Accessing your Fargate container logs on LOGIQ

Now that your Fargate container logs have been configured to be forwarded to LOGIQ, you can head over to your LOGIQ UI and access the Logs tab.

You’ll now see your Fargate container logs being ingested into LOGIQ.

Conclusion

f you’re not monitoring your ECS Fargate container logs on LOGIQ yet, now’s the time to get started. The flexibility that LOGIQ’s custom AWS FireLens Fluent Bit image provides enables you to seamlessly forward your Fargate container logs to LOGIQ andunify them with logs and metrics across all of your services and infrastructure. LOGIQ enables you to set up a unified log analysis stack for your engineering teams and lets them analyse logs and metrics side-by-side. If you’re not using LOGIQ yet, sign up for a free trial today.