Repository files navigation

ANuGet server with linux-first approach.

There seems to be no good nuget server for hosting private nuget packages and caching,when working mainly with linux and dotnet core.Running windows just to host a several nuget packages seems like a big waste.

This project aims at following:

• provideself-hosted nuget server for private package hosting.
• provide caching mode for public packages from nuget.org
• hosted with kestrel on dotnet core 2.0
• released as ready to use,end-to-end testeddocker image, preferably to be deployed to kubernetes.
• continuously tested with paket including several common project setup cases
• good performance when server is used by multiple clients,such as CI agents building various projects, downloading lots of packages at the same time.
• easy to develop on linux in VS Code, not only in VS on windows.

TL;DR since1.0.0 LiGet is a fork of BaGet. Read lower why...

We have previously created and used LiGet from various pojects, just to get it working on dotnet core.WhenBaGet started to look promissing,we contributed some work there with indention to migrate from LiGet to BaGet and obsolete the project.However, following was deal-breaker:

• What we consider critical basis for mature projectwas not merged:
  • build must be reproducible, which in current .Net world meanspaket.lock commited in source repository.
  • released product must be built CD-style. Which in short means to build artifacts only once, and run them through a pipeline of tests and QA. It is not acceptable to rundotnet build ordotnet publish several times for same commit. There must be a well-defined set of binaries which were tested through all pipeline stages.
  • if docker is released then docker image must be tested with end-case tests running actual nuget clients.
• long feedback time for PRs in BaGet. I spend only a few days at time to get job done. I cannot wait weeks for review.

How is this fork different from upstream BaGet:

• using FAKE for build system, rather than scripting in MsBuild.
• added unit, integration tests and e2e tests with paket and nuget cli.
• we use docker andDojo to create consistent, reproducibledevelopment environment for LiGet.
• added release cycle and testing of docker image using continuous delivery practices.
• implements read-through cache as a separate endpoint. Which at the timedoes not work upstream.
• uses paket and FAKE for build system.
• usesCarter for routing rather than bare Asp routing.
• adds ability to log to graylog
• adds V2 implementation from old LiGet
• caching proxy has different endpoint/api/cache/v3/index.json than private packages/api/v3/index.json

• Implements light-weightnuget V3 API for hosting private packages.
• Supports V3 search.
• LimitedNuGet V2 API for hosting private packages. Includes endpointsFindPackagesById(),Packages() andPUT /api/v2.Which is sufficient for clients to download, push, find or restore packages.
• Caching proxy of with NuGet V3 API.
  • Allows to cache.nupkg packages on server, rather than downloading them from the Internet each time.

Not implemented:

• Does not work with older paket versions
• Cache package metadata and invalidates when upstream changes are detected usingNuGet.CatalogReader.This will be ported from older liget < 1.0.0.
• V2 search, filter and alike queries.This is not planned.
• search and autocomplete endpoints for cached packages. Basically, you need to query nuget.org tosearch for public packages.

Seereleases to get docker image version.

docker run -ti -p 9011:9011 tomzo/liget:<version>

For persistent data, you should mountvolumes:

• /data/simple2 contains pushed private packages
• /data/ef.sqlite contains sqlite database
• /cache/simple2 contains cached public packages

You should change the default api key (NUGET-SERVER-API-KEY) used for pushing packages,by setting SHA256 intoLIGET_API_KEY_HASH environment variable. You can generate it withecho -n 'my-secret' | sha256sum.

Fordotnet CLI and nuget you need to configure nuget config~/.nuget/NuGet/NuGet.Config with something like:

<?xml version="1.0" encoding="utf-8"?><configuration>  <packageSources>    <addkey="nuget"value="https://api.nuget.org/v3/index.json"protocolVersion="3" />    <addkey="liget"value="http://liget:9011/api/v3/index.json"protocolVersion="3" />  </packageSources></configuration>

For paket, inpaket.dependencies, just specify another source:

source http://liget:9011/api/v3/index.json

dotnet nuget push mypackage.1.0.0.nupkg --source http://liget:9011/api/v2/package --api-key NUGET-SERVER-API-KEY

Fordotnet CLI and nuget you need to configure nuget config~/.nuget/NuGet/NuGet.Config with something like:

<?xml version="1.0" encoding="utf-8"?><configuration>  <packageSources>    <addkey="liget"value="http://liget:9011/api/cache/v3/index.json"protocolVersion="3" />    <addkey="liget"value="http://liget:9011/api/v3/index.json"protocolVersion="3" />  </packageSources></configuration>

For paket, inpaket.dependencies, just specify liget as the 2 only sources

source http://liget:9011/api/cache/v3/index.json# public packages...source http://liget:9011/api/v3/index.json# private packages...

If you have been using BaGet before, then many of your nuget sources in projects,could look like this, e.g. in paket:

source http://my-nuget.com/cache/v3/index.json# public packages (only in ai-traders fork)source http://my-nuget.com/v3# private packages

Above endpoints end up inpaket.lock too.LiGet has different endpoints (with/api before endpoints).If you want to deploy LiGet in place of BaGet and (at least temporarily) keep above endpoints,you can enable BaGet compatibity mode in LiGet.

LIGET_BAGET_COMPAT_ENABLED=true

This will enable following behavior:

• /cache/v3/index.json returns same content as our fork's BaGet's/api/cache/v3/index.json. Upstream BaGet does not have separate endpoint for public packages anyway.
• /v2/* returnsV2 resources, same as/api/v2/*

To make transition from old (<1.0.0) LiGet or any other server which keeps.nupkg files in a directory,there is animport command:

dotnet LiGet.dll import --path dir

In the docker image you can setup environment variable -LIGET_IMPORT_ON_BOOT=/data/simplewhich will cause liget to first search fornupkg files in$LIGET_IMPORT_ON_BOOT, before starting server.Packages which were already added are skipped.SettingLIGET_IMPORT_ON_BOOT=/data/simple is sufficient for migration from (<1.0.0) LiGet.

Note: you only need to set this variable once to perform initial migration.You should unset it in later deployments to avoid uncessary scanning.

The simplest start command is

mkdir -p /tmp/liget-testdocker run -p 9011:9011 -v /tmp/liget-test/:/data tomzo/liget

• Default current directory/data.
• Main process starts withtini as root,then drops privileges to run asliget user withdotnet.

For best reference see thedocker directory with Dockerfile and startup script.

All packages, cache, and temporary data is stored in /data. By default in/data/<backend>.

/data will be always owned byliget. Startup script switches uid/gid at startto fit with whatever was mounted from the host.The exception to this is when/data is owned byroot, then liget has to run asroot.``

Internally .NET Core application is configured by/app/appsettings.json file.But we are generating this file from environment variables to make it more user friendly.

Everything can be configured with environment variables:

• LIGET_SKIP_APPCONFIG_GEN - if true then/app/appsettings.json will not be generated on container start and you should mount your own. Default isfalse.
• LIGET_SKIP_RUNTIMECONFIG_GEN - if true then/app/LiGet.runtimeconfig.json will not be generated on container start and you should mount your own. Default isfalse.
• LIGET_API_KEY_HASH - hash of api key used for pushing packages.
• LIGET_EF_RUN_MIGRATIONS - runs entity framework database migrations on container start. Default istrue.
• LIGET_DB_TYPE=Sqlite - the type of database to use. Currently onlySqlite.
• LIGET_DB_CONNECTION_STRING - for sqlite, that is the path to db file. Default is"Data Source=/data/ef.sqlite/sqlite.db"
• LIGET_STORAGE_BACKEND by defaultsimple2. In1.0.0 introduced as the only implementation, replacing previoussimple.
• LIGET_SIMPLE2_ROOT_PATH - root directory used bysimple2 backend. By default/data/simple2.
• LIGET_SEARCH_PROVIDER - how to execute searches. By defaultDatabase.
• LIGET_BAGET_COMPAT_ENABLED - shouldBaGet compatibity mode be enabled. Default isfalse.
• LIGET_LISTEN_PORT - port where HTTP server will listen. By default9011.

Every dotnet Core application has.runtimeconfig.json, which can configure garbage collector.You may want to set following:

• LIGET_GC_CONCURRENT - by defaulttrue
• LIGET_GC_SERVER - by defaulttrue, beware though thatthis may cause higher memory use.
• LIGET_THREAD_POOL_MIN - minimal number of worker threads. By default 16.
• LIGET_THREAD_POOL_MAX - minimal number of worker threads. By default 32.

Kestrel specific:

• LIGET_LIBUV_THREAD_COUNT - number of libuv threads handling the requests. By default not set, determined by libuv default.

• LIGET_CACHE_ENABLED - default is true.
• LIGET_CACHE_PROXY_SOURCE_INDEX - address of original V3 API to cache. By defaulthttps://api.nuget.org/v3/index.json.
• LIGET_NUPKG_CACHE_BACKEND - backend of the .nupkg caching proxy. By defaultsimple2,which in1.0.0 was introduced as the only implementation.
• LIGET_NUPKG_CACHE_SIMPLE2_ROOT_PATH - root directory where dbreeze will store cached packages.By default/cache/simple2.
• LIGET_CACHE_INVALIDATION_CHECK_PERIOD - defines frequency at which a check with upstream server is made to see if cache is invalid. By default60 (seconds).Not Implemented yet in 1.0.0

• LIGET_LOG_LEVEL - by defaultWarning. Can beDebug,Information,Warning,Error.
• LIGET_LOG_BACKEND - by defaultconsole. Also can begelf.

LiGet is usingGELF provider for Microsoft.Extensions.Loggingto optionally configure logging via GELF to graylog.To configure docker image for logging to your graylog, you can set following environment variables:

• LIGET_LOG_GELF_HOST - no default. But should be configured whenLIGET_LOG_BACKEND=gelf
• LIGET_LOG_GELF_PORT - by default12201.
• LIGET_LOG_GELF_SOURCE - by defaultliget.
• LIGET_LOG_GELF_ENVIRONMENT - allows to set additional field in logged messages. By default not set.

We rely heavily on docker to create reproducible development environment.This allows to execute entire build process on any machine which has:

• local docker daemon
• docker-compose
• dojo executable on path. It is aCLI toolwrapper around docker and docker-compose which deals with issues such as ownership of files,mounting proper volumes, cleanup, etc.

You can execute entire build from scratch to e2e tests (liketravis).

• Install docker daemon if you haven't already
• Install docker-compose
• InstallDojo

DOJO_VERSION=0.4.0sudo wget -O=/usr/bin/dojo https://github.com/ai-traders/dojo/releases/download/${DOJO_VERSION}/dojo_linux_amd64sudo chmod +x /usr/bin/dojo

Then to execute entire build:

./tasks.sh all

This will pulldotnet-dojodocker image whichhas all build and test dependencies: dotnet SDK, mono, paket CLI, FAKE, Node.js.

Usage of Dojo is optional and you can easily contribute if you have above tools installed on your machine.

Releases are automated from the master branch, executed by GoCD pipeline, release is published only if all tests have passed.Travis executes the same tasks in the same environment and is for reference to the public community.If there is- Unreleased note at the top ofChangelog,then release is a preview, tagged as<version>-<short-commit-sha>.Otherwise it is a full release, tagged as<version>.

1. Fork and create branch.
2. Commit your changes.
3. Submit a PR, travis will run all tests.
4. Address issues in the review and build failures.
5. Before merge rebase on mastergit rebase -i master and possibly squash some of the commits.

If you have an idea or found a bug, open an issue to discuss it.

Firsly, this project is using lots of code from other nuget servers,either as reference or actually porting pieces of code.Credits:

• Loïc Sharma for creatingBaGet
• TanukiSharp/MinimalNugetServer as minimal dotnet core setup
• emresenturk/NetCoreNugetServer another minimal dotnet core server
• official NuGet.Server
• NuGet.Lucene which is part of klondike

This project is licenced under MIT.