What is Concourse? Learn the basics in 4 steps
Only 4 steps? Yes, for REAL.
Ok, I know how it feels, you are curious or just need to learn about this tool because of your work, and you need it fast. Well, friend, I would do my best and wrap it up in 4 steps:
1-Install fly
2-Create your own pipeline
3- Run your pipeline
4- Handle your pipeline
Step 0 — The beginning
The first time you hear about Concourse? Let me tell you this: is NOT like Jenkins. Just to be clear, you are going to read “is very easy to use” “ is better than…”. And maybe that is true, BUT at the beginning is hard to get, so my advise? Be patient because by the end is the truth and once you understand it is not hard to deal with.
What is concourse?
Concourse is an open-source continuous thing-doer.
Basically you are able to create, pipelines, jobs, tasks ( as in any other CI/CD tool) using just a simple YAML file for configure. You can configure it under Linux, mac or windows. And also it will allow you to create complex pipelines configuration if you want to know what I mean:
Step1 — Install Fly
One thing you should better know soon about Concourse
Concourse is primarily driven from the command-line; there is no GUI config wizard.
For that reason you will need to get Fly in order to handle commands to Concourse:
#Install fly on your local . Linux based
install fly /usr/local/bin#Check Fly version
fly -v
Step 2- Create your own pipeline
Now the time when you have to think a bit about what you really need on your pipeline. Do you need just 2 steps? 4 steps? which are the dependencies between each step? there is any step that can be run in parallel? You will need to answer all those questions, and more before be sure ( or at least have some initial proposal) to which pipeline you need to create.
I personally recommend begin easy, I mean 1 or 2 basic steps, which 1 dependence, just in order to you to understand how Concourse works, and then you can move on and add fancy stuff, more dependence and create an entire spaghetti ( if that is what you need, of course)
For that, you will need to create a YAML file which will contain the configuration setup for your pipeline ( let’s call this file pipeline.yaml)
##Example 1 single job
jobs:
- name: job-hello-world
public: true
plan:
- task: hello-world
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path: echo
args: [hello world]##Example 2 multiple jobs
---
resources:
- name: resource-tutorial
type: git
source:
uri: https://github.com/starkandwayne/concourse-tutorial.git
branch: develop
- name: resource-gist
type: git
source:
branch: master
uri: ((publishing-outputs-gist-uri))
private_key: ((publishing-outputs-private-key))jobs:
- name: job-bump-date
serial: true
plan:
- get: resource-tutorial
- get: resource-gist
- task: bump-timestamp-file
config:
platform: linux
image_resource:
type: docker-image
source: {repository: starkandwayne/concourse}
inputs:
- name: resource-tutorial
- name: resource-gist
outputs:
- name: updated-gist
run:
path: resource-tutorial/tutorials/basic/publishing-outputs/bump-timestamp-file.sh
- put: resource-gist
params: {repository: updated-gist}- name: job-show-date
plan:
- get: resource-tutorial
- get: resource-gist
passed: [job-bump-date]
trigger: true
- task: show-date
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: resource-gist
run:
path: cat
args: [resource-gist/bumpme]
So now you have the configuration file you can move forward and validate it, set it up and more! Look the following:
##Login to Fly
#fly -t {target} login -c {youConcourseURL } -n {yourTeamName}
fly -t target login -c https://concourse.url/ -n myTeam##Validate pipeline file
fly validate-pipeline -c pipeline.yaml##Set new pipeline or update existing one
#fly -t {target} set-pipeline -p {pipelineName}-c {pipelineFile.yaml}fly -t target set-pipeline -p pipeline-name -c pipeline.yaml##Check the pipeline state
fly -t target pipelines
Step 3- Run your pipeline
Once you set the pipeline, then you are going to be able to see it on the main concourse console:
Here, for example, I have running concourse on my local( that is why you are seeing localhost on the URL) and I have 3 pipelines created.
What do you want to do next? trigger a job for sure! And for that you can do it using the console or using the UI :
Instead, if you prefer the console :
##Trigger a job from a pipeline
fly -t {target} trigger-job --job pipelineName/jobNameThis will have the same effect, in both cases what you are going to see is a new build for your pipeline:
Related to run the pipeline I want to add some extra stuff here about how to cancel a build execution and how that is shown on the UI. For canceling you can only do it from the UI (Please if someone knows how to stop a build execution from the command line I will really appreciate that information!)
Let say you trigger a new execution, and then you realize that you introduce a mistake and also let’s say that the execution takes more than 1 hour, you don’t want to wait, just cancel and trigger a new build right? In this example, there is a build running #5 and if you click on the X , you will stop it (bear in mind here that any data, any lock on any resource will be unstable after this operation) and therefore you will see this:
As you can see the color for this case is brown and you will have at the end a label Interrupted
Step 4- Handle your pipeline
So you now know how to create the pipeline file, set it, running it… and now what? Next is the maintenance stage, you will need to be able to handle your pipeline.
To talk a bit about this, let me show you a really simple example for multiple jobs :
Regardless is a simple example, you can see here 3 things you can do on pipelines ( and in fact, you will NEED to do on pipelines) :
1-Input resources to you job execution ( in this example git resource)
2-Lock the resources (so no other job use the same resource you are using )
3-Passing params between jobs (in this case let’s say that the first job do some tasks and create information that is needed from the second job )
Some more things you can do are:
- You can as I mentioned before share params between jobs
- You can also define constraints/pre conditions for job execution
- You can integrate with tools for reporting as Slack
- You can determine trigger conditions ( by time or by another event)
- You can enter to the build and run stuff from there ( this can easily be another full article)
Nice/Useful commands !!
## Pause a job from a pipeline
fly -t {target} pause-job --j pipelineName/jobName##Unpause a job from a pipeline
fly -t {target} unpause-job -j pipelineName/jobName##Delete a pipeline
fly -t {target} destroy-pipeline -p pipelineName##Get pipeline from remote
fly -t {target} get-pipeline -p pipeline-name > path/localFile.yaml##List jobs for pipeline
fly -t target jobs -p pipeline-name##Enter to some job instance
fly -t {target} hijack -u https://yourUrl/pipelineName/jobs/JobName/builds/BuildNumber
More stuff/links
Last words
Hope you enjoy it!
I have to admit at the begging it was hard for me to understand concourse, and all the things I read always mention: “Ohh is so easy to use”. And YES, after you learn the basic and execute and play with it, I can assure is pretty straightforward.
See you next time!
