Apache RocketMQ

Apache RocketMQ stands as a comprehensive distributed message broker and data flow processing system, offering exceptionally low latency, superior throughput, proven dependability, capabilities reaching trillions of messages, and highly adaptable scaling features.

Key functionalities encompass:

Support for diverse messaging paradigms: publish/subscribe, request/reply interaction, and continuous data streaming.
Enterprise-grade transactional messaging guarantees.
Intrinsic resilience and high-availability frameworks leveraging the DLedger consensus mechanism.
Integrated message observability via internal tracing systems, with compatibility for OpenTracing standards.
Extensive interoperability with established big-data processing pipelines and analytical frameworks.
Message history retrieval based on either timestamp or sequential offset.
Guaranteed strict First-In-First-Out (FIFO) ordering and overall message sequence integrity within a single queue construct.
Optimized messaging consumption mechanisms supporting both client-driven pull and broker-driven push models.
Capacity to sustain millions of messages stored concurrently within an individual message queue.
Compatibility across multiple communication interfaces, including gRPC, MQTT, JMS, and the OpenMessaging specification.
Architecture designed for horizontal partitioning and seamless load distribution.
Extremely high-speed exchange of batched messages.
Sophisticated message filtering mechanisms utilizing SQL expressions and user-defined Tags.
Pre-packaged Docker container images facilitating isolated execution and cloud-native deployment.
A comprehensive operational console providing administration, performance metrics visualization, and system monitoring.
Robust mechanisms for access control, including authentication and authorization protocols.
A suite of open-source connectors facilitating reliable data ingestion (sources) and output (sinks).
Lightweight processing capabilities for real-time data manipulation.

Initial Setup Guide

This section delineates the procedures for deploying RocketMQ across various environments. For typical local testing setups, a singular instance of each required component will suffice.

Local Development Execution

RocketMQ operates across all primary operating systems, necessitating the presence of Java Development Kit (JDK) version 8 or a more recent iteration. Verification via the command java -version is recommended:

$ java -version
java version "1.8.0_121"

For users on the Windows operating system, please obtain the binary distribution package version 4.9.3 here: https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip. Unzip the contents to a preferred local directory, for instance, D:\rocketmq.

macOS and Linux users should execute the subsequent shell commands:

# Retrieve the distribution package from an Apache mirror site
$ wget https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip

# Extract the archive contents
$ unzip rocketmq-all-4.9.3-bin-release.zip

Open a terminal session, navigate into the extracted directory's bin subdirectory:

$ cd rocketmq-4.9.3/bin

1) Launching the NameServer Component

The NameServer will bind to the address 0.0.0.0:9876. Ensure this network port is unreserved on your local host. Initiation proceeds as follows:

For macOS and Linux environments:

### Initiate Name Server process in the background
$ nohup sh mqnamesrv &

### Verify successful startup by monitoring the log file
$ tail -f ~/logs/rocketmqlogs/namesrv.log
The Name Server boot success...

Windows users must first configure system environment variables: - Right-click the 'Computer' icon on the desktop and select 'Properties'. - Navigate to 'Advanced system settings'. - Select 'Environment Variables'. - Create a new System Variable named ROCKETMQ_HOME with the path set to your installation root, e.g., D:\rocketmq.

Subsequently, change the directory to the RocketMQ root and execute the Windows script:

$ mqnamesrv.cmd
The Name Server boot success...

2) Initiating the Broker Service

For macOS and Linux users:

### Start the Broker process, linking it to the NameServer
$ nohup sh bin/mqbroker -n localhost:9876 &

### Confirm Broker startup status (e.g., if Broker IP is 192.168.1.2 and name is broker-a)
$ tail -f ~/logs/rocketmqlogs/broker.log
The broker[broker-a, 192.168.1.2:10911] boot success...

For Windows users:

$ mqbroker.cmd -n localhost:9876
The broker[broker-a, 192.168.1.2:10911] boot success...

Deployment via Docker Containers

RocketMQ deployments can be readily containerized using Docker, utilizing the host network mode to correctly map container ports to the host machine.

1) NameServer Container Startup

$ docker run -it --net=host apache/rocketmq ./mqnamesrv

2) Broker Container Startup

Persistent storage for message data is mapped from the host's /tmp/store directory to the container target /home/rocketmq/store.

$ docker run -it --net=host --mount source=/tmp/store,target=/home/rocketmq/store apache/rocketmq ./mqbroker -n localhost:9876

Orchestration using Kubernetes

For large-scale, resilient deployments, running a RocketMQ cluster within a Kubernetes environment is facilitated via the RocketMQ Operator. Ensure that kubectl is correctly configured with the appropriate kubeconfig file prior to proceeding.

1) Installation of Custom Resource Definitions (CRDs)

### Clone the operator repository
$ git clone https://github.com/apache/rocketmq-operator
$ cd rocketmq-operator && make deploy

### Verify CRD installation success
$ kubectl get crd | grep rocketmq.apache.org
brokers.rocketmq.apache.org                 2022-05-12T09:23:18Z
consoles.rocketmq.apache.org                2022-05-12T09:23:19Z
nameservices.rocketmq.apache.org            2022-05-12T09:23:18Z
topictransfers.rocketmq.apache.org          2022-05-12T09:23:19Z

### Check the operational status of the operator pod
$ kubectl get pods | grep rocketmq-operator
rocketmq-operator-6f65c77c49-8hwmj   1/1     Running   0          93s

2) Instantiating the Cluster Resource

### Deploy the RocketMQ cluster definition
$ cd example && kubectl create -f rocketmq_v1alpha1_rocketmq_cluster.yaml

### Monitor the state of cluster components (StatefulSets)
$ kubectl get sts
NAME                 READY   AGE
broker-0-master      1/1     107m
broker-0-replica-1   1/1     107m
name-service         1/1     107m

Apache RocketMQ Ecosystem

RocketMQ Streams: A minimal framework for stream processing built atop Apache RocketMQ.
RocketMQ Flink: The official Apache Flink connector, providing stream and table API source/sink capabilities for RocketMQ data.
Multi-Language Client Implementations
RocketMQ Spring Integration: Facilitates rapid integration of Apache RocketMQ services within Spring Boot applications.
RocketMQ Exporter for Prometheus: A utility for exporting RocketMQ operational metrics for Prometheus monitoring.
RocketMQ Operator for Kubernetes: Simplifies the lifecycle management of Apache RocketMQ clusters deployed on Kubernetes.
RocketMQ Docker Images Repository: Source repository for official Apache RocketMQ container images.
RocketMQ Management Console: Web-based interface for system administration, metric visualization, and operational oversight.
RocketMQ Connect Framework: A framework designed for scalable and fault-tolerant data piping between RocketMQ and external data repositories.
RocketMQ MQTT Gateway: A specialized service implementing the MQTT protocol to handle message ingress/egress from IoT devices and mobile applications.
Incubating Community Projects: Projects currently undergoing incubation within the community, including tools like logappender, rocketmq-ansible, rocketmq-beats-integration, rocketmq-cloudevents-binding, etc.
RocketMQ Project Website Source: The source code repository for the official Apache RocketMQ documentation website.

Engagement and Support Channels

Discussion Forums: https://rocketmq.apache.org/about/contact/
Official Portal: https://rocketmq.apache.org
Documentation Hub: https://rocketmq.apache.org/docs/quick-start/
Issue Tracker: https://github.com/apache/rocketmq/issues
Improvement Proposals (RIPs): https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal
Q&A Platform: https://stackoverflow.com/questions/tagged/rocketmq
Real-time Chat: https://rocketmq-invite-automation.herokuapp.com/ (Slack Invitation Link)

How to Contribute

Contributions of any scope are highly valued, ranging from minor code refinement to the implementation of major new features or other forms of support. Detailed contribution guidelines are available here.

Licensing

This software is distributed under the terms of the Apache License, Version 2.0. Copyright (C) The Apache Software Foundation.

Export Regulations Notice

This software package incorporates cryptographic modules. Individuals or organizations must verify local jurisdictional regulations concerning the import, custody, utilization, and subsequent re-export of encryption technologies. Prior to deployment, consult applicable national laws and policies regarding encryption software. Refer to http://www.wassenaar.org/ for international guidelines.

The United States Department of Commerce, specifically the Bureau of Industry and Security (BIS), has classified this software under Export Commodity Control Number (ECCN) 5D002.C.1, applicable to information security software employing asymmetric cryptographic algorithms. This specific distribution format qualifies for export under the License Exception ENC Technology Software Unrestricted (TSU), as defined in BIS Export Administration Regulations Section 740.13, covering both compiled and source code forms.

The cryptographic functionality within this software is provided primarily via Apache Commons Crypto (https://commons.apache.org/proper/commons-crypto/), utilized for secure network communication authentication, encryption, and decryption services.

WIKIPEDIA INSIGHT: Cloud computing represents an "IT delivery model characterized by ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal administrative overhead or service provider interaction," as defined by ISO standards. This concept is colloquially known as "the cloud."

== Defining Characteristics == In 2011, the U.S. National Institute of Standards and Technology (NIST) established five core attributes essential to cloud computing environments. The precise NIST definitions are:

On-demand self-service: "A purchaser of computing capacity can provision processing, storage, and network capabilities unilaterally and automatically as required, without necessitating direct human intervention from the service provider for each request." Broad network access: "Capabilities are accessible across a network, utilizing standardized protocols and mechanisms that support a wide array of client devices (e.g., mobile devices, tablets, laptops, desktops). Resource pooling: " The vendor's aggregate computing assets are shared among numerous clients in a multi-tenant architecture, with resources being dynamically allocated and reallocated based on fluctuating consumer load." Rapid elasticity: "Provisioning and de-provisioning of capabilities can occur quickly, sometimes automatically, allowing scaling to match demand swiftly, both up and down. For the end-user, these resources often present as virtually infinite and immediately available." Measured service: "Resource consumption (including processing time, storage volume, data transfer rates, and active user counts) is automatically tracked, controlled, and reported via an abstraction layer suitable for the service type, ensuring clear transparency for both the supplier and the consumer." By 2023, the International Organization for Standardization (ISO) had updated and expanded upon this foundational list.

== Historical Context ==

The foundational concepts underpinning cloud computing trace back to the 1960s through the proliferation of time-sharing systems and remote job entry (RJE). The centralized 'data center' model, where users submitted batch jobs to dedicated system operators for execution on mainframe hardware, dominated this era. This period was defined by research into optimizing large-scale computational power distribution to a broader user base via shared access, focusing on efficient utilization of infrastructure, platform layers, and applications for enhanced end-user productivity. The term 'cloud' metaphorically representing virtualized services first appeared in 1994, employed by General Magic to describe the abstract space accessible by their mobile software agents within the Telescript framework. This metaphorical usage is credited to David Hoffman, a communications specialist at General Magic, building upon its established use in telecommunications and network diagrams. The phrase 'cloud computing' gained significant traction in 1996 following the internal business planning activities at Compaq Computer Corporation concerning the future of computing and internet services. The company aimed to revolutionize access to computation through the internet.