Where to find .gitattributes on Windows 10/11 using Git, IntelliJ 2022 and WSL Ubuntu to fix CRLF (\r\n) command not found: convert to LF line endings on checkout

Introduction

Problem: when checking out a project in IntelliJ in Windows, all files are checked out with Window's newline CRLF (\r\n).

But if you then open a terminal in IntelliJ which runs WSL (Ubuntu) and you want to run a bash script like this shell script, you'll get this error:

#!/bin/sh

set -e

[unrelated stuff deleted]

It will fail with: 

./deploy.sh: line 2: $'\r': command not found

: invalid optione 3: set: -

set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]

Those errors are caused by the shell script having CRLF instead of just LF, which Ubuntu/Mac OS expects.

Sidenote: I made softlink from /bin/sh to bash in my system because dash does not support several features, see here. 

So I tried setting IntelliJ's Code Style to use \n for newlines and line endings and line separator, also for new projects like mentioned here.

But still after creating a new project from Git, all files including the above script was set to CRLF. You can see this at the bottom of the screen in this screenshot:

I also realised that I don't want all files to have just LF, because I noticed doing that manually, on my commit of those files, I had to commit these too, as the newline changed. But I didn't want to bother other teammembers on Mac Books with this unnecessary change. Similar to what happens when you do a dos2unix on the file.

So the next to try was to get Git on my machine to handle it correctly. And thus hopefully IntelliJ too.

The .gitattributes file seemed a very good candidate: a suggestion was to change the git config 'core.autocrlf'. But that meant all the local files were getting changed still if I understood correctly, which I don't want.

Solution

It cost a lot of effort to find out where the gitattributes file is currently; and I wanted to change it for all my Git commands in the future, not just for one project.

What I wanted to change it to is mentioned here:

# Convert to LF line endings on checkout.

*.sh text eol=lf

**/*.sh eol=lf

Those lines specify to change all end of lines (newlines) of files ending with .sh to be checked out with a LF (\n).

I also added the 3rd line myself, to specify subdirectories, but maybe that was not needed.

It was hard to find the correct location for the gitattributes (or .gitattributes file which made it all more confusing), this official Git text wasn't super-clear on it: 

This guy also had a challenge on finding their location.

Finally I found my gitattributes file on Windows here:

C:\git\etc\gitattributes

C:\git is actually where I installed the Git client. And it is the one IntelliJ uses (but not the one the terminal shell in IntelliJ uses!) 

And there were already quite a few settings in there like:

*.RTF diff=astextplain

*.doc diff=astextplain

*.DOC diff=astextplain

I just appended these:

**/*.sh eol=lf

*.sh eol=lf

Restarted IntelliJ to be sure. And yes, after a complete full new checkout of my Git project, Java and Kotlin files were still having CRLF line endings, and my shell script had the LF ending.

Docker build with Git command running in CircleCI failing with: Fatal: No names found, cannot describe anything, invalid argument, for "-t, --tag" flag: invalid reference format

Introduction

Context: Docker, CircleCI, Github.

The Docker build command 

docker build -f .circleci/Dockerfile -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:`git describe --tags` -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:${CIRCLE_BUILD_NUM} -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:${CIRCLE_SHA1} .

was failing with this message:

fatal: No names found, cannot describe anything.

invalid argument "********************************************/my-project:" for "-t, --tag" flag: invalid reference format

See 'docker build --help'.

Exited with code exit status 125

Solution

You'd expect the Docker command maybe being syntactically incorrect. But the error message is referring to something else: it turns out the git describe --tags command gave the fatal message.

The cause was that there was no git-tag set at all on the Github project yet.   After manually creating a release (including a tag) on Github and running the above build command again, the docker build command succeeded.

Gitlab maven:3-jdk-8 Cannot get the revision information from the scm repository, cannot run program "git" in directory error=2, No such file or directory

Introduction

In a Gitlab project setup the most recent Docker image for maven was always retrieved from the internet before each run by having specified image: maven:3-jdk-8 in the gitlab-ci.yml. The image details can be found here.

Of course this is not a best-practice; your build can suddenly start failing at a certain point because an update to the image might have something changed internally causing things to fail.

What you want is controlled updates. That way you can anticipate on builds failing and plan the upgrades in your schedule.

The issue and workarounds/solutions

And indeed suddenly on March 29 2018 our builds started failing with this error:

[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create (useLastCommittedRevision) on project abc: Cannot get the revision information from the scm repository :
[ERROR] Exception while executing SCM command.: Error while executing command. Error while executing process. Cannot run program "git" (in directory "/builds/xyz"): error=2, No such file or directory

That message is quite unclear: is git missing? Or is the directory wrong? Or could the maven buildnumber plugin not find the SCM repository?

After lots of investigation it turned out the maven:3-jdk-8 image indeed had changed about 18 hours before.

And after running the maven command in a local version of that Docker image indeed the same error occured!  Awesome, the error was reproducable.

And after installing git again in the image with:

- apt-get update
- apt-get install git -y

the error disappeared!  But a new one appeared:

[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

This also hadn't happened before. After some searching it turned out it might be the surefire and failsafe plugins being outdated.

So I updated them to 2.21.0 and indeed the build succeeded.

Here's the issue reported in the Docker Maven github. UPDATE: it is caused by an openjdk issue (on which the maven:3-jdk-8 is based upon.

This issue made us realize we really need an internal Docker repository. And so we implemented that :)

One disadvantage about Docker images is that you can't specify a commit hash to use. Yes you can specify a digest instead of a tag, but that is a unique UUID hashcode only. You can't see from that hashcode anymore the (related) tagname.

Lessons learned - Jackson, API design, Kafka

Introduction

This blogpost describes a bunch of lessons learned during a recent project I worked on.

They are just a bunch grouped together, too small to "deserve" their own separate post :)

Items discussed are: Jackson, JSON, API design, Kafka, Git.

Lessons learned

• Pretty print (nicely format) JSON in a Linux shell prompt:
  
  cat file.json | jq
  
  You might have to 'apt-get install jq' first.
  
  
• OpenOffice/LibreOffice formulas:
  
  To increase date and year by one from cell H1031: =DATE(YEAR(H1031)+1; MONTH(H1031)+1; DAY(H1031))
  
  Count how many times a range of cells A2:A4501 has the value 1: =COUNTIF(A2:A4501; 1)
  
  
• For monitoring a system a split between a health-status page and performance details is handy. The first one is to show issues/metrics that would require immediate action. Performance is for informational purposes, and usually does not require immediate action.
  
  
• API design: Even if you have a simple method that just returns a date (string) for example, always return JSON (and not just a String of that value). Usueful for backwards compatibility: more fields can be easily added later.
  
  
• When upgrading gitlab, it (gitlab) had changed a repository named 'users' to 'users0'. Turns out 'users' is a reserved repository name in gitlab since version 8.15.
  
  To change your local git settings to the new users0 perform these steps to update your remote origin:
  
  # check current setting
  $ git remote -vorigin  https://gitlab.local/gitlab/backend/users (fetch)
  origin  https://gitlab.local/gitlab/backend/users (push)
  
  # change it to the new one
  $ git remote set-url origin https://gitlab.local/gitlab/backend/users0
  
  # see it got changed
  $ git remote -v
  origin  https://gitlab.local/gitlab/backend/users0 (fetch)
  origin  https://gitlab.local/gitlab/backend/users0 (push)
  
  
• Jackson JSON generating (serializing): probably a good practice is to not use @JsonInclude(JsonInclude.Include.NON_EMPTY)  or NON_NULLm since that would mean a key will be just not in the JSON when its value is empty or null. That could be confusing to the caller: sometimes it's there sometimes not.  So just leave it in, so it will be set to null.   Unless it would be a totally unrelated field like: amount and currency. If amount is null, currency (maybe) doesn't make sense, so then it could be left out.
  
  
• Java:
  
  Comparator userComparator = (o1, o2)->o1.getCreated().compareTo(o2.getCreated());
  
  can be replaced now in Java 8 by:
  
  Comparator userComparator = Comparator.comparing(UserByPhoneNumber::getCreated);
  
  
• Kafka partitioning tips: http://blog.rocana.com/kafkas-defaultpartitioner-and-byte-arrays
  
  
• Kafka vs RabbitMQ:
  
  - Kafka is optimized for producers producing lots of data (batch-oriented producers) and consumers that are usually slower that the producers.
  - Performance: Rabbit: makes about 20K/s  Kafka: up to 150K/s.
  - Unlike other message system, Kafka brokers are stateless. This means that the consumer has to maintain how much it has consumed.
  

Best of this Week Summary 09 November - 15 November 2009

• A post on experiences with Git coming from Subversion. Too bad the reasons why the author is having more and more problems with Subversion were not given. Though you can deduct (some(?) of) them from the text.
  


• Presentation by John Resig (from JQuery fame) on testing javascript. Shortly describes QUnit, JSUnit, YUITest for unittesting, Screw.Unit, JSSpec for behavior testing (unittesting broken up by task), Selenium for functional testing and Crosscheck, Env.js, Blueridge for server-side Javascript testing. Also describes browser launching, which I would translate into "build/test management", distributed testing (Selenium, TestSwarm) and browser support (cost vs benefit).
  


• "A Decade of SOA: Where are we, Where are we Going?" The viewpoints from a virtual panel of Enterprise Architects who have lived and implemented SOA for most of this decade.
  


• Why iStockphoto embraced Google's Gears.
  


• And as grand final this week: the hardware and (rails) software architecture of Ravelry, a knit and crochet community.
  

Best of this Week Summary 03 November - 09 November 2008

• Three reasons why you could switch from Subversion to Git (distributed source code management tool, created by Linux Torvalds). Here's a quite extensive comparison and here you can compare it yourself.
  


• What makes good REST? Roy Fielding lists a couple of criteria.
  


• Tips and Q&A on optimizing and tuning Apache Tomcat.
  


• Yahoo! opened up its open strategy platform: Yahoo! Application Platform (YAP), Yahoo! Social Platform (YSP), and Yahoo! Query Language (YQL). All services are accessed via oAuth.
  


• Is the Sun (pun intended ;-) setting for Swing in favor of JavaFX???